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Abstract. Building on recent work of Kowalski on random walks on SL{n, Z) and Sp{2g, Z), 
we consider similar problems (we try to estimate the probability with which, after k steps, the 
matrix obtained has a characteristic polynomial with maximal Galois group or has no nonzero 
squares among its entries) for more general classes of sets: in GL{n,A), where A is a subring 
of Q containing Z that we specify, we perform a random walk on the set of matrices with fixed 
determinant D € . We also investigate the case where the set involved is any of the two 
left cosets of the special orthogonal group SO{n,m){Z) with respect to the spinorial kernel 
n(n,m)(Z). 



Introduction and statement of the results 



For G a fixed subgroup of GL{n, Q), it is natural to wonder what the typical behavior of an 
element g G G chosen at random should be. That kind of question is investigated by Kowalski 
E"' ' in [KoSievei Chap. 7]. With in mind such intuitive facts as: a random element should have, 

^ . with high probability, an irreducible characteristic polynomial (or indeed, one with large splitting 

(-^ ! field) and no square among its entries, Kowalski shows that the k-th step of a random walk lies 

in the set of the exceptional elements of G (i.e. the elements which do not satisfy the desired 
property) with probability tending to zero exponentially as k grows to infinity. 

In loc. cit., these results are obtained, in the case where G = SL{n,Z) or Sp{2g,Z) (for 
n ^ 2 and g ^ 2), as an application of the very general large sieve framework exposed in the 
, first chapters of jKoSievej . 

■ In this paper, we answer the same type of questions (i.e. we try to detect similar properties) 

! for sets Y being either left cosets aSL{n, A) of GL(n, A) (where n ^ 2 and ^ is a subring of Q 

containing Z which we will specify) or left cosets of SO{n, m)(Z) for n+m ^ 6 (i.e. we will fix an 
indefinite quadratic form with signature {n,m) when seen as defined over a (n + m)-dimensional 
space over R) with respect to the normal subgroup Q{n,m){Z) (which is to be described later). 
The method used is that of the "coset sieve" described by Kowalski in [KoSievel Chap. 3.3] 
(see also [KoZeta| where that idea already appears to study properties of the numerator of zeta 
functions of curves over finite fields). 



Let us now define what is needed to give the precise statements for the main results of this 
\ paper. The first kind of subgroups G of GL{n, Q) we consider are of the type G = GL{n, A) 

where, if V denotes a (possibly infinite) set of primes with complement having positive density, 
then A is taken to be equal to the ring Z[1/'P] which is the smallest subring of Q containing Z 
in which every p G V is invertible. The left coset to which we apply large sieve techniques in 
that first case is a fixed element of GL{n, A)/ SL{n, A). 

We also consider the case where G is the subgroup of integral points of a special orthogo- 
nal group: for n + m ^ 6, let {M,Q) be a quadratic module over Z such that, seen over R, 
the quadratic form Q is indefinite with signature {n,m). The group of automorphisms of M 
preserving Q can be seen as the subgroup of integral points (denoted 0(n, m)(Z)) of the alge- 
braic group 0(n, m)/Q. We will restrict ourselves to the case where G = SO{n,m){Z), the 
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subgroup of integral points of the algebraic group SO{n,m)/Q. In SO{n,m){Z) lies the nor- 
mal subgroup 0(n,m)(Z) (see |HMl Section 7.2C, pp 422-424] where that subgroup is denoted 
0'{M)). A precise description for that group will be given in Section [2] in the case where M 
is a vector space over a finite field and in Section 13.11 in the general case. However, to state 
our results, the important thing is that the fixed coset we consider in that case is an element of 
SO{n,m){Z)/n{n,m){Z) (an abelian quotient; see f HM l. 7.2.21]). 

In the sequel, we emphasize the case where (Af, Q) is a free hyperbolic module over Z (see [HMl 
page 197]), i.e. M is a Z-module of rank 2n equipped with a quadratic form Q (with attached 
bilinear form denoted h) such that there exists a basis of isotropic vectors X = {xi, . . . ,X2n) 
such that, 

Id 



Id 



where the inner blocks are n x n matrices. 

Seen over R, such a quadratic form has signature (n, n) and we will restrict ourselves to such 
quadratic forms to state Theorem [1] (which is a sample of Theorem [17] in which the case of more 
general quadratic modules is handled). 

The question of the irreducibility of the characteristic polynomial of a an element chosen 
"at random", in one of the two types of groups we have just described, is only relevant if no 
trivial factorisation pattern is imposed by the definition of the groups involved. If we only 
suppose that g G GL{n,A), there is no a priori imposed factor for the characteristic polynomial 
Pg{T) = det(T — (7), but things are different if g is an orthogonal matrix. Indeed, Pg verifies in 
that case the functional equation: 

(1) p,(r) = det(-5)r^p,(i), 

where g is assumed to be a x matrix. 

It seems natural now to wonder about the factorisation properties of the reduced characteristic 
polynomial which is defined by 



det(r - g)r. 



ed 



det{T - g)/{T -det{g)), if N is odd, 
det(T - g)/(T2 - 1) , if A^ is even and det{g) 
det(T — g) , otherwise . 



Here, the matrix g will always lie in the special orthogonal group attached to Q, so that, in 
the case where A^ is even, we will always have det{T — g)red = det{T — g). Notice moreover that 
the degree Nred of det(T — g)red is always even. 

Now, with the above notation, let G be the group GL{n, A), for n ^ 2 (resp. SO{n, ^^)(Z), for 
n ^ 3), and G^ the normal subgroup SL{n,A) (resp. Q{n,n){Z)) of G. Let S be a symmetric 
generating system for G^ (i.e. for any s G 5, we have € S). Notice here that we do not 
assume G^ to be finitely generated, so that S could be infinite (but still countable). Let {ps)ses 
be a sequence of strictly positive real numbers indexed by S satisfying X^^ggPs = 1 and ps = Pg-^ 
for any s € 5. Finally let a be a fixed element of G. 

Suppose a probability space (^',S,P) is given and let {Xk)k be the (left invariant) random 
walk on the left coset aG^ defined by 

Xq = a , Xk+i = Xkik+i , 

where {Ck)k^i is a sequence of independent uniformly distributed random variables with values 
in S and law 

P{^k = s)=P{Ck = S-^)=Ps, 

for any s € 5. 

Our main result quantifies the speed of rarefaction of "non-typical" elements reached by the 
fe-th step of the random walk as k grows. In order to state it in a unified way, the reduced 
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polynomial det(T — g)red denotes, in the first case, nothing but the usual characteristic polyno- 
mial det(r — g); while in the second case, the ring A denotes nothing but the ring Z. A "weak" 
version of our result can be stated as follows: 

Theorem 1. With notation as above, there exists a Pi > such that for all 1, we have 

P(det(T — Xi^)red £ ^[T] is reducible) <^ exp(— , 

with /?! depending only on the underlying algebraic group G/Q, on the generating set S and on 
the sequence {ps)s (i-e-- on the distribution of the £,k)- Moreover the implied constant depends 
only on G and the density of V in the set of all rational prime numbers (in the case where 
G = GL{n,Z[l/V])). 

There exists a (32 > such that for all k 1, we have 

P(an entry of the matrix is a square in A) <^ exp(— /32fc) , 

with the same dependency for P2 ds for /3i and the same dependency for the implied constant as 
in the previous case. 

In the above statement, the underlying algebraic group is SL(n) if G = GL(n, Z[l/7-']) (with 
n ^ 3) and SO(n, n) if G = SO{n, n)(Z) (with n ^ 3). 

Of course, the second statement of Theorem [T] is only a very special case of the kind of 
properties that can be investigated using larege sieve techniques. In Section [3l we give a more 
general statement of the above Theorem in which the common points of the properties that are 
likely to be successfully studied via our method appear clearly. 

1. Estimates for the large sieve constants 

In this section we will often need to refer to results coming from the large sieve techniques 
exposed in the appendix. So, before getting into the proof of Theorem [U the reader might 
either want to check the appendix or assume Propositions [25] and [26] (which are self-contained 
statements) to be true and postpone the reading of the whole appendix. 

With notation as in Propositions [25] and [26l let A be a set consisting of odd primes (with the 
additional condition A n P = in the case where G = GL{n, Z[l/7'])) and C* the finite subset 
of A consisting of the elements smaller than a given integer L ^ 1. 

In both Propositions [25] and [26] (note that it is natural to emphasize the case where the sets 
are conjugacy invariant as the estimates are improved when using this special property), the 
heart of the large sieve method lies in the following inequality 

(2) P(p£(Xfc) ^ for all£ ^ L) ^ A{Xk,L)H-^ , 

where 

H = Y,\Qim\-mr' 

is the saving factor which depends only on the sieving sets 0^ and where we denote A(Xfc, L) for 
the constant A of Propostions [25] and [26] with the above choice of C*. In this section, though, 
we focus on the large sieve constant A(Xfc,L) for which we give an upper bound in the case 
where G is one of the two groups that Theorem [1] deals with. 

The possibility to obtain the sort of quantitative information stated in Theorem [1] for the 
random walk (Xk) defined in the introduction depends crucially on the sharpness of the upper 
bound we can find for the large sieve constants involved. It is not realistic to hope for any useful 
explicit bound without any assumption on the group G we are working with. As the sums (fTTjl 
and ([To]) involve representations of the group G (that factor through finite groups), the fact that 
Lubtotzky's Property (r) comes into play is not so surprising. Let us first review some definitons 
and facts concerning that property. 
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1.1. Lubotzky's Property (r). We first recall the definition of the property, as stated in |LZl 
1.4] or (Lul Page 49]. 



Definition 2. Let G be a topological group and M = {Ni}i a family of normal finite index 
subgroups of G, indexed by a set I. The group G has Property (r) with respect to M if there 
exists a finite set S and e > such that, for any unitary irreducible continuous representation p 
of G on a Hilbert space Ti such that ker(/)) D Ni for some i and that leaves no nonzero vector 
invariant, we have 



for all nonzero v & H. The pair (e, S) is called a (r)-constant for G and S is called a (r)-set for 



If, in the definition, we do not require p to factor through a subgroup taken from a fixed 
family, then the group G is said to have Kazhdan's Property (T). So, as is obvious from the 
definition, Lubotzky's Property is a "weak" version of Kazhdan's. That means, of course, that 
any group with Property (T) also has Property (r) with respect to any family of its finite index 
subgroups. However, that property is indeed strictly weaker: for instance 5L(2,Z) does not 
have (T) (see [HVl Prop. 6 page 34]) but has (r) with respect to the family of its congruence 
subgroups 



That last fact, though, does not come for free, as it requires Selberg's result on the eigenvalues 
of the hyperbolic laplacian acting on L^(r(d)\H) (see, for instance [Lul 4.4]). 

Remark. A useful interpretation for Property (r) is the following: if G is a group and S is a 
subset of G, recall that the Cayley graph C{G, S) is the oriented graph with vertex set equal to 
the set of elements in G and where x is connected to y if there exists an s G S such that xs = y. 
If we suppose that S is symmetric, then C(G, S) can be considered as a non oriented graph and 
if S spans G, that graph is connected. Now, with these two additional assumptions, the fact 
that G has Property (r) with resepct to a family M = {Ni}i of subgroups is equivalent to the 
property of expansion of the family of Cayley graphs {C{G/Ni, Si))i, where, for each i, Si is the 
projection of S to the corresponding quotient. More generally a graph X = {V, E) is said to 
be a 5-expander graph (where <5 > 0), if, for any subset A of V containing less than one half of 
the elements of V, the number of vertices in V \ A which are neighbors of elements of A is at 
least S\A\. Moreover, the expansion ration 6 is explicitely related to the (r)-constant for G with 
respect to Af. The notion of expander, born in the 1970's in order to solve problems linked to 
networks, has motivated lots of mathematical researches. The beautiful constructions of such 
families that can be found in |Chung| , |Mo| or [LPS], rely heavily on deep mathematical tools. 

If G is finitely generated (which is the case in the applications developped by Kowalski 
in [KoSievei Chap. 7]) and has Property (r) with respect to M, it can be shown that any 
generating set S can be chosen as the (r)-set for G (see [LZl Prop. 1.2] or [Lul Th. 4.3.2]). 
For the applications we have in mind, however, we need to work with groups which are not 
necessarily finitely generated (this is obviously the case for SL{n, Z[l/V]) if V is infinite). The 
following result, explained by M. Burger, shows that in this case, we can also choose a (r)-set 
among the elements of a generating system for G. 

Proposition 3. Let G be a group with Property (r) with respect to a family of finite index 
subgroups Af = {Ni}i. Let S be a generating system for G, then there exists a finite subset Sq 
of S which is a {T)-set for G. 

Proof. Let F be a (finite) (r)-set for G and (5 > such that {5, F) is a (r)-constant for G. As F 
is finite, there exists a subset 5o of S and an integer n ^ 1 such that F C Sq (i.e. each element 
in F can be written as the product of at most n elements of 5o). Now let vr : G ^ ^{H) be 
a continuous unitary representation of G which factors through Ni for some i (i.e. kervr D Ni) 



max ||/9(s)t; — v\\ > £\\v 



G. 



r{d) = kei{pd : SL{2, Z) ^ 5L(2, Z/dZ)) 
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and without any nonzero invariant vectors in 7Y. li v G 7i has norm 1, then there exists 



/ 



Sq G F such that 



S^Mf)iv)-v\\ 



Using the fact that the representation vr is unitary, the right hand side of the above inequahty 
can be written 



n— 1 



j=0 



J+1 



){v) -ir{sl,---sl)v 



n-1 



j=0 
n-1 

j=0 



Combining these last two series of inequahties, we deduce there exists atoGSo such that 
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n 



^ ||vr(to)(v) - v\ 



□ 



Before explaining how Property (r) yields the kind of upper bound we need for the large 
sieve constants, let us first give (an infinite family of) examples of groups having Property (r) 
(with respect to a certain family of subgroups for each of these examples). Note moreover that 
in the case where n = 2 the following lemma provides us with infinitely many examples of 
groups having Property (r) (with respect to a suitably chosen family of subgroups) without 
being Kazhdan groups. These groups, of course, are directly involved in the proof of Theorem [H 

Lemma 4. Let V be a proper subset of the rational primes. For any n ^ 2 the group SL{n, Z[l/V]) 
has Property (r) with respect to the family of its congruence subgroups 



kevTTd : SL{n,Z[l/V]) SL{n,Z[l/V]/dZ[l/r] 



{d^l\p^d ifp&V} 



Proof. Let 5*1 be a finite generating system for SL{n,Z) (the elementary transformations for 
instance). As already mentioned, 5i is a (r)-set for 5L(n,Z). The natural inclusion 

SL{n,Z) ^ SL{n,Z[l/V]) 

enables us to consider a generating set S D Si for SL{n, Z[l/P]). Let m be an integer without 
prime factors in V; we consider the projection 

7r„ : SL{n, Z[l/V]) SL{n, Z[l/V]/mZ[l/r]) ^ SL{n, Z/mZ). 

The restriction of the morphism -Km to SL{n, Z) is surjective, so, since the family of Cayley 

graphs ((^(^^(n, Z)/(ker TTm n 5L(n, Z)), 7rm('S'i)) ) (indexed by the integers m coprime to any 

element in V) is an expander family (see the discussion preceding remark above and recall that, 
for n ^ 3, the group SL{n, Z) is a Kazhdan group), then so is the family 



(c{SL{n, Z[l/V])/ ker vr^, 7r^(Si)) 



A fortiori, the family [C{SL{n,Z[l/'P])/ ker 7rm,T^m{S))] forms of family of expanders. In 

other words, the group SL{n, Z[l/V]) has Property (r) with respect to the family of its congru- 
ence subgroups. □ 
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1.2. Upper bounds for A(Xfc,L). Coming back to our sieve framework (see the appendix), 
we now state the key proposition making precise all the assumptions that will need to be verified 
for the arithmetic groups we consider, in order to obtain a sufficiently sharp bound for the large 
sieve constants A and H of Propositions [251 and [261 (this is the analogue of [KoSievel Prop 7.2]). 

Proposition 5. We recall G is a discrete group, a normal subgroup ofG with abelian quotient 
G/G^ ant let T he a finite subset of G/G^ in which we let a vary. For a fixed symmetric 
generating system S of G^ we consider the random walk (Xk), defined in the introduction, on 
the coset aG^ (with a G T) and we suppose 

• there exists a relation of odd length c among the elements of S: 

Si • • • Sc = 1 , 

• the steps E,k, for k ^ 1, are independent and independent of Xq, 

• G^ has Property (r) with respect to a family (A'j)jg/ of finite index subgroups, 

then there exists r/ > such that, for any finite dimensional representation 

TT-.G^ GL{V) , 

satisfying kervrigg D Ni, for some i and without any nonzero G^ -invariant vector, the inequality 

|E((^(Xfc)e;/))| ^ ||e||||/||exp(-7?A:), 

holds for all vectors e, f in V and all ^ 0; ( ; ) denoting a G -invariant inner product on V. 

The constant rj only depends on the {T)-constant associated to (G^, S, {Ni)), on the distribution 
of the ^fc cind on the length c of a fixed relation in S. 

Proof. The proof is quite similar to that of [KoSievel Prop. 7.2]; many technical points need to 
be modified, though, so we give the full detail of the arguments here. 

We fix an index i G / and a representation 

TT : G ^ GL{V) , 

such that the restriction n^Qg factors through Ni and has no nonzero G^-invariant vector. Con- 
sider 

M = E(7r(efc)) = ^p(s)7r(s); 

M is a well-defined element of End(F) since the series defining M converges absolutely (because 
TT is a unitary representation and YlsP(^) ~ M, we can then define two other elements 

of End(F): 

M+ = Id - M , M- =ld + M. 

Note that these formulae define two operators which are both independent of k and self 
adjoint. Indeed, the set S as well as the distribution of the are symmetric; moreover the 
mapping associating its adjoint to an operator is linear and continuous. We also need to define 

iVo = B{tt{Xo)) = 5^P(Xo = t)7r(t) G End(y) . 

The random variables Xq and being independent, for /c ^ 1, we have 

B{7r{Xk)) = N^M^ , 

thus, by linearity, 

E((7r(Xfe)e;/)) = (M'=e;iV*/), 
where Nq denotes the adjoint of Nq. 

YlisesP^^) ~ ^ since for every s G 5, 7r(s) is a unitary operator, the eigenvalues of M 
are in the interval [—1; 1]. Now, let p be the spectral radius of M: 

p = max{|7| I 7 is an eigenvalue ofM} . 

6 



We have the inequahty 

|(M^e;iVo*/)l^l|e|l 11/11/, 

since the norm of A^o is smaller than 1. 

We need to exhibit a <5 > independent of i and vr such that ^ p ^ 1 — 6. We will then 
be able to choose r] = — log(l — 5) > for the constant we are looking for. We use the fact 
that p = max(p+,/9^) where p'^ (resp. p~) is the real number equal to the greatest positive 
eigenvalue of M (resp. equal to the opposite of the smallest negative eigenvalue of M). It is 
enough to prove that p^ < 1 — 6± for some constants S± which are independent of i and vr. To 
that purpose, we use the variational interpretation for the eigenvalues of a self adjoint operator 
on a finite dimensional Hilbert space. Indeed 1 — p~^ (resp. 1 + p~), which is the smallest 
eigenvalue of (resp. of M~), is equal to 

. {Tv; v) 
A = mm ^ , 

where T = M^. Now applying Proposition [3l we know that there exists, for the group , a 
finite (r)-set Sq included in 5; this yields 

{M+v]v) _lsr^ \\tt{s)v-v\\'^ 
f r 2 ^-^ \\v\r 

II M ^gg II II 



1 ||7r(s)t' — v\ 



2^'' \\v 

seSo 

^ mi mi max ■ 



2 v^o seSo ll'^IP ' 

where = miiis^So Pi^) > ^ runs over the respresentations of without any nonzero 
G^-invariant vector and which factorize through Nj for some index j. Let k > be such that 
(k, 5o) is a (r)-constant for G^ with respect to (A'j)jg/, we can choose 



6+ 



2 

To determine 6~ , the argument is very close to that of |KoSieve[ Prop. 7.2]. As there exists, 
by assumption, a relation of odd length c among the elements of S, we write, for v & V, 

V = ^(^{v + tt{si)v) - {it{si)v + 7r(siS2)^^) H 1- (7r(si • • • Sc-l)v + TT{l)v)j . 

Then, invoking Cauchy-Schwarz's inequality and using the G-invariance of the inner product, 

c-1 c-1 

|f 11^ ^ 4 lk(ri)t; + TT{riSi+i)vf ^ ^ 11^ + 7r(si+i)f ||^ , 

i=0 i=0 

where rg = 1 and rj = si • • • Sj for i ^ 1. In particular, we deduce 

1 ''""^ 

^ iiW"cK^g^(^^+^) lb + vr(.,^.i).|^ 
then, taking into account the possible repetitions of generators in the sequence (si, . . . , Sc), 



^ — — — ; 7 p(s) ||7r(s)w + wlp ^ — ( min I 1 ^ i ^ c| | (M v:v) . 

" 4 min{p(si) | 1 < i ^ c} ^ n w n ^ 2 V i^v v i ^ ^ \ > / 

Therefore we can choose 6~ = min {p{si) | 1 ^ i ^ c} > 0. □ 

The next two propositions give, under the assumptions of Proposition [5] together with an 
additional hypothesis of linear disjointness (which really is a property of "independence of i of 
the setting") the upper bound we need for the two large sieve constants A we are working with. 
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To begin with, we consider the case of the conjugacy coset sieve which is somewhat simpler 
to handle. 

Proposition 6. Let {Y,A,{p£ : Y Yi)) be the conjugacy coset sieve of the appendix (see 
Proposition ^^) . We suppose that 

• the assumptions of Proposition are verified, 

• the system (/j^)^gA is linearly disjoint, i.e. the restricted product map defined for I., I' G A, 

I + e, by 

Pi/' =PiX : ^ Gl,, = Gf X Gl 

is surjective. 

Then, with notation as in Proposition \2B, there exists t] > such that 

A{Xk,L) ^ 1 + L^exp(-??A:), 

where t] > depends only on G, S and the distribution of the cmd A = {3d + 2)/2, with d = 
n? — I in the case where G = SL{n, Z[l/7^]) and d = (n + m)(n + m — l)/2 if G = SO{n, m)(Z). 

Proof. Prom Proposition [25], we have 

A(X/£, L) ^ max max ^ ^ |Ty(7r,r)|. 

^ e'ec* Ten*, 

For i, i' £ A, we need to give an upper bound for the sums 

W{v^,^r) = , ^ E{Tt[Tr,f]pe^e'{Xk)) , 
If? I If:, I 



once rewritten using the trick explained in the appendix after Lemma [24l 

With notation as in the appendix (see the paragraphs before and after Lemma [24l). if [tt, f]p£/' 
has no G^-invariant vector then Proposition [5] yields an upper bound for 

|E(TV[^,f]p,,,,(Xfc))|. 

Indeed, it is enough to choose e = / running over an orthonormal basis of the representation 
space V of [vr, f]p£^£' and then to sum the terms obtained over e. 

We are reduced to computing the multiplicity of the trivial representation of G^ in the restric- 
tion of [vr, f]p£^i' to G^ . As the sieve setting we work with is supposed to be linearly disjoint, that 
quantity is the same as the multiplicity of the trivial representation of G^^, in [vr,f]|(j9 . From 

Lemma [21] of the appendix, we know that multiplicity is zero unless (^, vr) = {i',T) in which 
case its value is |r^|. Thus, denoting [vr,f]o the part of [vr,f] without any nonzero G^-invariant 
vector, we deduce: 

Tr[7r,r]p„,(Xfc) = \tj\6{{e,7r), {£' ,t)) +TT[Tr,f]ope,AXk) ■ 
Applying Proposition [5] to the representation [7r,f]o/0£/' of G^ yields 

\W{7r,T) -6{{i,7r),{£',T))\ ^ (dim^)(dimT) exp(-??A;) , 



-1 



where we use the trivial upper bound [^y ir^llrplj ^ 1. 
The result follows from exploiting such trivial bounds as: 

dimyr ^ ^/\G\ , ^ dimvr |G| , 

7rGirr(G) 

for any irreducible complex representation tt of a finite group G (see |KoSieve[ Chap. 5] for 
better bounds for such quantities). 

□ 



In the next proposition, we give an upper bound for A(Xfc, £*) in the case of the non-conjugacy 
coset sieve, very close to the one stated in Proposition [6l However, needing to use another 
equivalence relation to define the orthonormal basis for the space involved (compare the 
statements of Proposition [251 and [26]) . the above proof cannot be directly adapted to the case of 
the non-conjugacy coset sieve. Indeed, to prove the following result, we use the remark following 
Proposition [26] about the generalisation of the sieve statements of the appendix to a framework 
in which we do not only use primes but more generally squarefree integers to perform the sieve. 

Proposition 7. Let {Y = aG^,A,{p£ : Y —f Yi)) be the non conjugacy coset sieve of the 
appendix (see, e.g., Proposition\2B\) . For any fixed integer L ^ 1 and under the same assumptions 
as in Proposition\B, there exists rj' > such that 

A{Xk,L) ^ 1 + L^' exp(-r/'A;) , 

where r]' > depends only on G, the {T)-constant for , the distribution of the o-nd A' = 
{I7d + 4)/4 with d = -1 xfG = SL{n,Z[l/V]) and d = {n + m){n + m - l)/2 if G = 
SO{n,m){Z). 

Proof. As in the proof of the previous proposition, we need to evaluate a sum involving group 
characters. The point is that the maximal contribution, in those sums, comes from the function 
corresponding to the trivial representation. Following that idea, we apply an "equidistribution 
approach" in order to obtain the estimate we are aiming at: 

E(([vr,r](/9^,|./(Xfc))e;/)[^,^]) = ^ ([vr, f](y)e ; , 



where the notation are those of ([15]) and where ly, on the right hand side of the equality is 
defined by: 

ly = P{pi^i'{Xk) = y) . 

To evaluate ly we decompose the characteristic function Xy of {u} in Fourier series. To 
that purpose, we need to extend by multiplicativity the result of Proposition [26] to the case of 
squarefree integers (not only primes, see the remark in the appendix, after Proposition [26]). In 
L^(y£^^', the following equality holds: 

Xy= ^ {V:Xy)'^= ^ '^{y)\GlA'^ip. 

Thus, we obtain 

ly = ^{xy{pi,t'Xk)) = iT^-l X] '/'(y)E(v?P«'(^fc)) 

= i7^ + T7^ ^ '/'(y)E(m/'(Xfc)) . 

Now, \i if = (/?7r^ ^,,e',/' is an element of S^^^/ (up to a suitable normalisation, see Lemma [23]) 
different from 1, we know in particular that vr^^^/ is an irreducible representation of Gi^ii and the 
quantity for which we need to find an upper bound is 

To apply Proposition [5] we need to determine the multiplicity of the trivial representation 
of in the restriction of vr^ ^/ to . As we assume linear disjointness of the sieve setting, 
this multiplicity is the same as that of lea in TTi^tPE,e for the group G^ . Applying Lemma [24] 
(once extended by multiplicativity) with vr = vr^,^', r = Iq^ (note that the assertion of Lemma [Ml 
remains valid for the trivial representation, see |KoSieve[ Proof of Lemma 3.2]), and using the fact 
that £ is a prime factor of ppcm(£, £'), we see that multiplicity is zero unless ppcm(£,/) = I (i.e. 
i' = £) and vr^ £',^g = Iiqo (or more precisely, applying IKoSieve[ Lemma 3.2], tt^ £'(^ip ~ Iq, „, 



for a certain character tp of Gi^i' /G^ ^,). In particular, tt£^i' has dimension 1 and, for every vector 
e' with norm 1 spanning the representation space of vr^ ^/ and every g E G^^,, 

{■^e,e'{9)e';e') = {e';e') = 1, 

where the index tti/^ is purposely omitted to avoid the use of too much notation. Thus, with 
notation as in Lemma [23l we deduce that (/^tt, ,,,e',e' ~ 1, which is a contradiction. 



Invoking proposition [5] now yields an ry' > such that for all ip = (p-K^^,,e'j' G Si/i \ {1}, 
Finally, for the quantity 

(dim7r)(-l/2)(dimr)(-^/2) W{ip^^e,f,^T,e,<p) - S{^7r,eJ,^r,e,4>) ' 

we obtain the following upper bound (note that the inverse of the denominator of the normali- 
sation factor is trivially smaller than 1): 

' s/ey,,,, veB,,,,\{i} 
Applying Cauchy-Shwarz's inequality, we obtain 

|([7r,f](y)e;/)[^^^]| ^ ||[vr,f](y)e||[^_^] ||/||[^,^] 
^ 1 

and more generally \(p{y)\ ^ 1, for all y £ Y^^p and all (p E Bi^p \ {!}. We deduce an upper 
bound for (dim7r)("^/2)(dimr)("^/2) W{<pn,ej,^T,e,<t>) ~ ^iv^n,e,f,<PT,e,<j)) , by using the triangle 
inequality, 

{\Be/' \ - l)exp(-r/'/c) . 
Now, by classical group representation theory, 

\Be,e'\ ^ Y (dimvr)^ = \Ge,e'\. 

7reirr(G^ 

We finally deduce an upper bound for the large sieve constant: 

A{Xk,L) ^ 1 + L^' exp(-r//c) , 

with A' = {17d + 4)/4 and either d = - 1 if G = GL{n, Z[l/V]), oi d = (n + m)(+m - l)/2 
if G = SO{n,m){Z). Indeed, from the argument above, we just need to use the same kind 
of trivial bounds as the ones at the end of the proof of Proposition [6] as well as the obvious 
inequality jG^/'l ^ 

□ 



2. Local densities for polynomials and orthogonal matrices 

In this section, which is independent of the others, we compute different densities in subsets 
of the ring F^[T] or of the orthogonal group 0(A^, F^) (notice that we do not assume anything 
here on the integer N and, in particular, we do not distinguish between the split and non split 
model for the orthogonal group, in the case N is even). 

The goal of this section is to give enough quantitative information in order to find a useful 
lower bound for the constant H appearing in ([2]). However, that section having an interest of its 
own, we do not restrict ourselves to the computations that are strictly needed for the purpose 
of the paper. The style in which we expose the different estimates we are interested in is very 
much inspired by |Chavl Section 3]. Doing so, it is easy to point out the common points as 
well as the differences between the symplectic case (treated by Chavdarov in loc. cit.) and the 
orthogonal case. We will namely highlight that the lack of good topological properties for the 
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orthogonal group imposes to be very careful in the statement of our results (such precautions 
need not be taken in the case of the symplectic group). 

2.1. Review of orthogonal groups over finite fields. We briefly recall some basic facts and 
notation about orthogonal groups, as exposed in |KaLl Section 6]. The proofs and details can 
be found e.g. in [ABS| . 

Let y be a vector space with dimension greater or equal to 2 over a fixed finite field with 
characteristic different from 2. We assume we are given a non degenerate quadratic form Q on 
V (we will denote by ^> the bilinear form attached to Q). If T(y) denotes the tensor algebra 
associated to V, we can consider the ideal '3{Q) generated by the elements x(g)x — (5(x).l, where 
X runs over the elements of V. The quotient algebra Cl{V,Q) = T{V)/J{Q) is the Clifford 
algebra of V with respect to Q. That construction yields a natural injection 

iQ-.V ^T{V)^Cl{V,Q), 

which enables us to see Cl{V,Q) as the solution of the following universal problem: for every 
morphism of Fg-vector spaces f : V where A is an Fg-algebra satisfying /(x)^ = Q{x)Aa, 

there exists a unique F^-algebra homomorphism / : Cl{V, Q) ^ A such that f oiq = f ■ 

Now, the involution v i— > —v in F, can be extended to an involution, denoted /, of Cl{y,Q). 
Another morphism plays a crucial role in the theory of Clifford algebras: it is the antiautomor- 
phism t : Cl{V,Q) Cl{V,Q) coming from the natural antiautomorphism defined on the fc-th 
tensor power of V by 

Vl ^ V2 ® ■ ■ ■ Vk '-^ Vk ^ ■ ■ ■ ® V2 ^ Vl . 

Let Cl^ be the group of invertible elements of CliV, Q). It acts on CliV, Q) via the morphism 
p defined by 

p{u)x = I{u)xu~^ , 

for every u G Cl^ and x G CliV^Q). The elements of Cl^ that leave V globally invariant form 
a subgroup, denoted and called the Clifford group, of Cl^ . Typical elements of are the 
images in Cl{V,Q) of non isotropic vectors v ^V, since the transformation x i— > I{u)xu~^ is 
then the reflexion with respect to the hyperplane which is orthogonal to v. In fact, any element 
of is a scalar multiple of a product of such vectors (the transformation associated to that 
element being an automorphism of the quadratic module {y,Q)). 

Finally, we define the map Norm : u G ^ t{u)u which takes its values in F^ (see (ABSl 
Prop. 3.3 and 3.8] where the proof, given in the case where the base field is R, can be easily 
adapted to the finite field case). The spinor group Spm{V,Q) is then defined as the subgroup 

Spin(l/, Q) = (kevNormy , 

of the elements of that are fixed by /. When V is A^-dimensional and there is no ambiguity 
on the chosen quadratic form Q, we will denote that group Spin(A^, Fg) instead of Spm{V,Q). 
With such notation, the group Spin(A'', Fg) can in fact be seen as the group of Fg-rational points 
of an algebraic group defined over Fg, denoted Spin(A^), and which we will also refer to as the 
spinor group. It is well-known that the spinor group is a connected simply-connected algebraic 
group and that it is in fact the universal cover of the special orthogonal group SO(A^)/Fg. In 
other words ( [Hul page 189]) there exists an isogeny ip such that we have an exact sequence of 
algebraic groups 

(3) 1 y {±1} > Spin(iV) SO(A^) > 1. 

Thus the spinor group shares the same dimension as the special orthogonal group N{N — 1)/2 
and the same rank [A^/2j. 

Remark. In all of the above, we do not need to assume that the base field is a finite field. Every 
construction and definition we have recalled can in fact be stated for quadratic modules over 
any perfect field. 
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Now, if Fq denotes a fixed separable closure of Fg, the short exact sequence ([3]) gives rise to 
the following exact sequence of Gal(Fg/Fg)-invariant groups: 

(4) 1 {±1} > Spin(iV,Fg) SO{N,Fg) {±1} ^1, 

where the group homomorphism Ngpin is called the spinor norm and can be defined as follows. 
For a non isotropic vector v £ V, the image of the reflection with respect to v by Ngpin is the 
class of in F^ /(F^)^ (the group of classes modulo nonzero squares of F^ ). The morphism 
Nspin is entirely determined by the images of those elements and it can be shown (see [KaLl 
Section 6]) that Nspin can be extended to a surjective morphism from 0{N, Fg) onto {±1} since 
we have supposed N ^ 2. 

Finally, we will denote Q{N,Fg) the image of p in Q (i.e. the kernel of Ngpin). That group 
which is of great importance in our sieving context, is easily seen to be the derived group of 
both SO{N,Fq) and 0{N, Fg) (see 5.17]). We note that it can be directly defined from 

0{N, Fg) by saying that it is the simultaneous kernel of the determinant and the spinor norm. 

To perform the density computations we need, we will have to estimate the cardinalty of 
sets of orthogonal matrices with fixed determinant and/or spinor norm. In practice, it will 
be convenient to relate those quantities to the number of polynomials with coefficients in the 
base field that can be realized as characteristic polynomials of such matrices. To exhibit the link 
between these cardinalities, the crucial point lies in the possibility to "see" the value of the spinor 
norm and of the determinant of a matrix g in the coefficients of its characteristic polynomial. 
This is, of course, very easy in the case of the determinant. But as far as the spinor norm is 
concerned, we do not see any obvious reason for such an explicit link to exist. However the 
following beautiful result of Zassenhaus (which we recall here, in view of its importance), gives 
us, under certain conditions, the kind of link we need: 

Theorem 8 (Zassenhaus, 1962). If g is an element of the orthogonal group associated to the 
quadratic space {V,Q) (satisfying the same assumptions as above), then, provided —1 is not an 
eigenvalue of g, we have: 

det(^) = Nspin(5), 
where both det and Ngpin are seen as applications with values in {±1}. 

In |Za| . Zassenhaus first defines the spinor norm via the formula of the theorem above (see 
(2.1) in loc. cit.) and then proves that definition coincides we the one we gave earlier in this 
section (see the corollary of [Zal Th. page 446]). 

Remark. Over finite fields of odd characteristic, we know that there are two isomorphism classes 
of quadratic forms (classified by the value of the discriminant of the quadratic form). In the 
case where the dimension is odd, these two classes give rise to the same orthogonal group, 
but this does not hold if the dimension is even. Indeed if A'^ = 2n, two distinct models for 
the orthogonal group 0(2n, Fg) need to be distinguished: they are respectively called the split 
and nonsplit model, refering to the algebraic group 0(2n) being split or not over Fg (see [KaLl 
Section 6] for examples of quadratic forms corresponding to each of these two models). Note that 
the computations in the sequel are performed independently of the chosen model of orthogonal 
group. However, we will see later how a result of Baeza makes the choice of the split or of the 
nonsplit model come back into play. 

2.2. Characteristic polynomials of orthogonal matrices over finite fields. Let £ be an 

odd prime number and ^ 2 an integer. As in the previous subsection, (V, Q) still denotes 
a quadratic space over F^ and Q is still assumed to be non degenerate. For g an element of 
0(A^, F^), we denote by Pg the reversed characteristic polynomial of g: 

Pg{T) = det{l-Tg). 

This polynomial also satisfies the functional equation ([T]). A short proof of that fact goes as 
follows: if a is the automorphism of the ambiant quadratic space {V, Q) attached to the matrix 
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g, we denote by Q the matrix of the quadratic form Q wrtitten in the basis in which the matrix 
of a equals g. Then we have gQi^g) = Q; we deduce 

Pg{T) = det(l - Tg) = det(l - T{'g)) 

= dei{l -T{Q-^g-^Q)) 

= r^det(<7-^)det(5/r - 1) = (-T)^ det(ff)P,(l/r) . 

Here one should be careful that Pg no longer designates the "usual" characteristic polynomial 
det(r — g), as in the introduction. However, it is easily seen (and it can be proved in the 
exact same way), that the reversed characteristic polynomial Pg still satisfies ([T]). Moreover, 
another motivation to change the notation is that Chavdarov works with reversed characteristic 
polynomials in [Chavj . so that we can easily understand to what extent his results can be 
transposed to the orthogonal case. 

The functional equation ([T]) imposes Pg to be "almost self-reciprocal", i.e., its roots cti, . . . , oat 
can be reordered in such a way that 

UiaN+i-i = 1 , 1 ^ i ^ n, if = 2n , 

aittN+i-i = 1 , 1 ^ -i ^ n and a^+i = det((7), if A'" = 2n + 1 , 
That leads naturally to the consideration of the set of polynomials 

Mn/ = {l + feiT + • • • + bNT^ I bi e Fe, b% = 1 and b^-i = bNh, if ^ f ^ [N/2\ } . 

Before studying certain subsets of M]\f^g, we recall a result due to Edawrds which gives in the 
case where is even a (quite surprising at first) link between the discriminant of a polynomial 
/ G Mjv,^ and the values / takes at ±1. 

Lemma 9 (Edwards). Let N be an even integer and let f £ Mjy^i be a monic separable polyno- 
mial, then we have 

disc(/) = /(!)/(-!) (mod (F;)2). 

That result is obtained by combinig Theorem 1 and Theorem 2 of |^ (note that the definition 
of the discriminant of a polynomial used in loc. cit. is not the standard one hence the statement 
of Lemma [9] only coincides with the one of [Ej up to sign). 

We are interested in the cardinality of certain subsets of M^^^. Our first result in this direction, 
which is very close to |Chavl Lemma 3.2], deals with the subset of irreducible polynomials in 
Mtv,^, or rather of those which are irreducible once reduced. Indeed, using the notation of the 
introduction and remembering we consider reversed characteristic polynomials here, we will use 
the notation 

• if = 2n, 

Kh,i = {fiT) G M^^e I f{T)red = is irreducible} , 



with e = lif6jv = — 1 and e = otherwise, 
if A^ = 2n + 1, 



Kh,i = {fiT) G MN,e I f{T)red = is irreducible} , 

with e = ±1 if bN = =fl. 
In both cases we will denote by Nj-ed the degree of the reduced polynomial fr^d- 

For those sets we have the following estimates: 
Proposition 10. * If N is even, 

1 + fN/2-e gN/2-6 
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// N IS odd, 



l,f{N-l)/2 f(N-l)/2 



Proof. It is enough to consider the case where N is even and e = 0. Indeed, if N is even and e = 1 
(which means that the leading coefficient of / is —1), we are interested in the irreducibility of 
g{T) = /(T)/(l — T^). The leading coefficient of g is obviously 1 and it is clear that g € M^_2/. 
Thus g G K^_2 ^ as soon as / G Kj^ ^. 

In the case where is odd, then g{T) = /(T)/(l— eT) has leading coefficient 1 and g G M^„i ^ 
so that g G -K'Jv-i e soon as / G Kf^ ^. In other words: 

• if is even, l-K^^^I = \K^_2^\, and 

• if is odd, \K%^'e\ = \K%_^^'g\. 

We are now reduced to computing the number of elements of 

^N,e — |/(^) ^ M]\r^£ I even , b^^i = ftj, ^ i ^ N/2 , / monic, irreducible | . 

This happens to be exactly the set Kg studied by Chavdarov in [Chavl Lemma 3.2] for 
g = N/2. In loc. cit., Chavdarov proves that for A^ even, 

and this completes the proof. □ 

Restricting ourselves to polynomials / satisfying / = fred (i-e. A^ = deg/ is even and / is 
monic), we are interested in the same kind of computation as above with the extra condition 
that / must be such that /(I) and /(—I) are in a given class (not necessarily the same for /(I) 
and /(—I)) of modulo its subgroup of nonzero squares, i.e., by LemmaOand Theorem[Hl we 
work with polynomials having fixed discriminant and that can only be characteristic polynomials 
for elements with fixed spinor norm. At first, it seems likely that we will get a "good" proportion 
of such polynomials among the set of irreducible self-recipocal polynomials of even degree (that 
proportion should roughly be 1/4). However, the following result of Meyn (see |Mel Th. 8]) tells 
us that this intuition is wrong: 

Proposition 11 (Meyn). Let f he a self-reciprocal monic polynomial of even degree N over F^. 
Let us write 

f = x^'^h{x + x-^) 

with h monic of degree N/2. If h is an irreducible polynomial then f is irreducible iff h{2)h{—2) 
is a nonsquare of F^ . 

Moreover, with notation as in the above satement, it is easy to see that if we start with an 
irreducible /, the attached polynomial h will also be irreducible. So any self-reciprocal monic 
polynomial / with /(I) and /(—I) chosen in such a way that (— l)'^/^/(l)/(— 1) is a square in 
F£ is not irreducible. This means that out of the four classes determined by the imposed value 
in 1 and —1 modulo squares, only two are non empty. 

The following result asserts that the expected equidistribution property holds for the two non 
empty classes. 

Lemma 12. Let N ^ A be an even integer. Then, if (non necessarily distinct) 



fixed elements of a fixed set {l,eo} of representatives o/F^/(F^)^ and if (— l)^/^e^ e 
nonsquare of Fi, then we have 

|/ G Mtv/ I / is irreducible and f{a) = e\ , f{b) = 



IS a 



2{l + N) 

^ 2N 



where, in the set of the left hand side, congruences are taken modulo the group of nonzero squares 
ofF,. 
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Proof. By Meyn's Theorem and the discussion preceding Lemma [T2l the set we are interested 
in is in one-to-one correspondence with 

|/i G Fi[T] monic of degree N/2 \ his irreducible and h{2) = ej^^ , h{-2) = {-l)'^/'^ef^^ . 

If we vary e^^^ and in {l,eo}, we see that computing the cardinahty of the above set 
amounts to evaluating the four following character sums 

\Y.{i±Mh{2mi±xdh{-2))), 

h 

where the sum is taken over all monic degree N/2 irreducible polynomials of F^lT] and where 
Xi denotes the Legendre character of F^. 

It is enough to focus on the study of the sum 

5 = (1 + X£(M2)))(1 + Xe{h{-2))) = 1+^ xK/i(2))+E xKM-2))+E Xi{h{2)h{-2)) 

h h h h h 

The first sum of the right hand side is nothing but the number of irreducible monic polynomials 
of degree N/2 in F£[r]. There are well known lower bounds for that quantity. For our purpose, 
it is enough to use the inequality (see, e.g. |Chavl Lemma 3.1]) 

y 2f^_^/^ 

^ N 

h 

Next we consider both the sums X]/iX^(^(2)) and X^(^(~2)). As N/2 ^ 2 (and thus an 
irreducible h of degree N/2 cannot be X it 2), we have, from the definition of xt- 

Y,Xe{h{2)) = 2 \{h\h{2) = a}\-Y,l 

h a nonzero square h 

Notice that the summand of the right hand side does not depend on the point of F^ at which h 
is evaluated (i.e. imposing the value of h at any point of F^ yields a set with the same cardinality 
as the analogue set where the value imposed is h{0)), so using on the one hand the lower bound 
(see [KoSievel Appendix B, formula B.8]) 

21^/2-1 

\{h G Fe[T] I h monic irreducible ,degh = N/2,h{2) = a}\ ^ — — i^''^ , 

and on the other hand the upper bound (see [Chavl Lemma 3.1] or |KoSievel Appendix B, 
Lemma B.l]) 

2£N/2 

\{h G Fe[T] j h monic irreducible ,degh = N/2}\ ^ ^ , 

we get: 

2£^/2 , ,/ 2^^/2-1 



E»(M2))<^-(.-i)(^-."") 

h 



nfN/2-1 
^ N ^ 

For the remaining sum, we need to be more careful (as the value of the polynomials considered 
at two different elements of F^ are involved). To begin with, the sum can be expressed as follows: 

xdh{2)h{-2)) = 1 X£(Norm((-l)^/2(2 - a)(2 + a))) , 

h a,dega=N/2 

where Norm denotes the norm map with respect to the extension F^iv/2/F^. Now, using the 
inclusion-exclusion principle, we get 

Y xKNorm(4 - a^)) = J2 (-1)^/'"' J2 xKNorm(4 - a^)) . 

a,dega=N/2 d\N/2 ot£F^i 
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For each divisor d of N/2, if we set xe,N = Xe ° Norm (which is a multiplicative character of 
F^d), we need to evaluate J2aeF a ^^.^(^ — a^). Prom the Riemann Hypothesis for curves over 
finite fields, we derive 

I y: - c.')\ ^ 

since the polynomial X'^ — 4 has distinct roots in F^ (see [KaMull Intro and Th. 1] for the state- 
ment and the proof of a more general result handling the case of higher dimensional varieties). 
Using the trivial fact that the number of divisors of N/2 is less than N/2, we get the upper 
bound: 

- xKNorm(4 - a')) ^ ^^^/^ . 

«,dega=Af/2 

Thus 

Yxi{h{2)h{-2)) -e^/\ 

h 

Finally, putting the above estimates together, we get 



N N N \ I 



□ 



In the last lemma of this subsection, we are interested in counting monic polynomials h of 
degree N/2 with certain imposed factorization patterns and imposed value modulo squares at 2 
and —2. Indeed it will be convenient, in the sections to come, to use such information in order 
to prove the existence of certain elements in the Galois group of an integral polynomial / whose 
reduction / (mod^) can be written / = x^/'^h{x + x~^). 

Lemma 13. Let N ^ A he an even integer and i ^ b he prime. With notation as ahove (e.g. all 
the congruences are taken modulo the suhgroup of nonzero squares ofFg), if we denote 

(i) Qi^3 for the set of monic polynomials f € M^^i such that the corresponding h is separahle, 

has an irreducihle factor of prime degree > N/A and such that h{2) = e^^\ h{—2) = sf^ , then 
we have 

,~ , £^/^/7 15iV, 

(a) ©£_4 for the set of monic polynomials f E Mjy^i such that the corresponding h is separahle, 
has a unique irreducihle quadratic factor, no other irreducihle factor of even degree and such that 
h{2) = e^l\ h{—2) = £'f \ then we have 

with an ahsolute implied constant. 

Proof, (i) Let be a prime such that N/A < £' ^ N/2. The cardinality we are computing is 
greater than that of the set of monic polynomials h of degree N/2 which factor as the product 
of a monic irreducible polynomial of degree £' with imposed values modulo squares at 2 and 
—2 with any monic irreducible polynomial of degree N/2 — £' (note that N/2 — £' < N/A so 
that no double root may occur in this way). So, applying Lemma \H\ (more precisely, using the 
arguments of the proof) and using once more the lower bound of [Chavl Lemma 3.1], we get 



4fV £ J\N/2-£ 

As N/A < £' ^ N/2, we have, on the one hand 

2(l + 2f) N ^ 4£^/^ / 2(1 + jV) 

if ~£ / N/2 - £' ^ N ' 



16 



and on the other hand 



^/ _ 2(l + 2f)x wv/4-^v2 < ^^^1 _ 2 + 

Gathering those two inequahties we get the estimate we wanted to estabhsh. 

(ii) We consider separately the case where N/2 is odd and the case where N/2 is even. In 
the former case, the cardinahty of G^^4 is greater than that of the set of polynomials factoring 
as the product of an irreducible quadratic polynomial having imposed values modulo squares at 
2 and —2 with any monic irreducible polynomial of degree N/2 — 2. Thus 



.N/2 - 2 

Now, if N/2 is even, the set we consider contains all the polynomials which, once divided 
by their quadratic factor (still with imposed values modulo squares at ±2) are products of a 
polynomial of degree 1 (different from X ±2) with any irreducible polynomial of degree N/2 — 3 
(note that if = 4, such a polynomial does not exist and, if = 8, the other factor of degree 
1 as well as the polynomials X ±2 must be removed from the set from which that polynomial 
of degree N/2 — 3 is picked). Thus, using the same inequalities as above, 

'i{N = A, |e,,4| ^ f fi-fV 

ifA^ = 8, |e,,4| ^ f U-f K^-2)(£-3), 

if AT ^ 12, |e,,4| ^ f (l - f ) (^ - 2) - , 

so that we get in particular the estimate stated. □ 

2.3. Number of matrices with prescribed characteristic polynomial. In our sieving 
context, we need a result which would be the analogue, in the orthogonal case, of [Chavl Th. 
3.5]. The point is that we need to know, for a fixed / G Mtv/, how many matrices in 0{N, F^) 
have a reversed characteristic polynomial equal to /. Towards such a computation, our first 
task is to show that there exists at least one matrix g S 0{N,Fi) such that Pg = f. This 
is, at least, a crucial step of the proof if we try to follow Chadarov's method. Unfortunately 
his proof relies heavily on the fact that the symplectic group Sp{2g) (as an algebraic group 
over F^) is simply connected and thus (by a Theorem of Steinberg), that the centralizer under 
Sp{2g) of any semisimple element in Sp{2g^Y^) is a connected algebraic group to which Lang's 
rationality Theorem may be applied. As we already mentioned neither O(A^) nor SO(A^) is 
a simply connected algebraic group. As a matter of fact, we can easily construct examples 
of polynomials in Mjv/ which are not the reversed characteristic polynomials of any matrix 
in 0(A^, F^). Take, e.g. the polynomial f{T) = + T + 1 G and suppose that the 

quadratic structure on the ambiant space F^ x F^ is given by the standard scalar product: 
$((xi, yi), (x2, 2/2)) = X1X2 + 2/12/2- A straightforward computation shows that any matrix in 
0(2, F^) with / as reversed characteristic polynomial must have its non diagonal coefficients 
equal to half a square root of 3, and this is obviously not always possible for matrices with 
coefficients in F^ (problems already occur for £ = 5...). 

While the direct adaptation of Chavadarov's method to the orthogonal case seems to be 
hopeless, we can however use a result of Baeza (see [Ba| ) that gives a very useful criterion, in 
the case where the dimension is even, to decide whether an / G Mat^ is or is not the reversed 
characteristic polynomial of a g G 0(A^, F^). Prom Baeza's result, we derive the following 
proposition: 

Proposition 14. Let N he even and f G M^/ such that 

(1) / is monic, 

(2) / is separable, 

(3) disc(/) = disc(Q). 
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Then there exists a semisimple element A € SO{N,Fi) such that 

det(l -TA) = f . 

The equality of discriminants (condition (3)) is seen as an equality of residue classes in 
/ (F^ )^. This condition is crucial and we easily see that, in our counterexample, disc(P) = —1 
while disc((5) = 1 in the case where £ = 5. 

Proof. Prom [Ba|, Th. (3.7)], we know that the quadratic forms Q' on V such that there exists 
a cj G SO{V,Q') satisfying 

det(l - Ta) = det{a - T) = f{T) , 

are exactly those that can be written s^{K,n), in the notation of loc. cit. We briefly recall how 
these quadratic spaces are constructed: the separable F^-algebra K = F^[T]/(/(r)) = F^[x] is 
equipped with the involution 

X I— > . 

If we consider the subalgebra L = Fi{x + then we have a norm map 

n: K ^ L, 

that defines a non degenerate quadratic form {K, n) with coefficients in L. For any trace map 
s : L — >• F^, we can then consider the composition s o n : K ^ Y f/\i defines a non degenerate 
quadratic form on K. We denote s^{K,n) the quadratic space obtained (see |Bal discussion 
following Prop. 3.6] and the references therein for details, namely concerning trace maps). 

For such a fixed quadratic space s*(i^, n), let us consider a o" € SO{s^:{K,n)) such that 

det(l - Ta) = det(cT - T) = /(T) . 

From |Bal Th. 1.2], we then have disc(/) = disc(s*(K, n)) thus, by assumption, (l\sc{s^:{K, n)) = 
d\sc{Q). But we know that quadratic forms over F^ are classified by their discriminant: so 
s^,{K,n) ~ {V,Q). Finally, to the element a corresponds a matrix A G SO{V,Q) ~ SO{N,F£) 
such that 

det(l - TA) = f{T) . 

The semisimplicity of A is obvious from the separability assumption on /. □ 

In that proof, we see how the distinction between the two models of orthogonal groups (in 
the case N is even) naturally appears. In particular we notice that, with the notation of Baeza, 
the quadratic space s<t:{K,n) is precisely the one chosen by Katz in [KaLl Section 6] to describe 
a model for the nonsplit orthogonal group. 

We are now ready to prove the main result of this section. Provided the assumptions of 
Proposition [14] are verified, the statement is the analogue, in the orthogonal case, of |Chavl Th. 
3.5] and it can be interpreted as a property of equidistribution of characteristic polynomials of 
orthogonal matrices among the polynomials of Mj^^i. Apart from the use of Proposition [TU the 
arguments developped in the proof are quite close to those of loc. cit. 

Theorem 15. Let N be even and let f G M^v/ be such that 

(1) / is monic, 

(2) / is separable, 

(3) disc(/) = disc(Q). 

Then, 

[b G 0{N, F,) I /(T) = det(l - Ti?)} 
with an implied constant independent of N. 
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Proof. Let A be a semisimple element of SO{N,F£) with reversed characteristic polynomial 
equal to / (the existence of such an A is justified by Proposition [T4l) . Let 

A = {B e 0{N, Fe) I det(l - TB) = /} . 

If S € A, its Jordan decomposition can be written 

B = BgBu , BgBu = BuBs , Bs, Bu € 0{N, F^) , 

with Bg semisimple and B^ unipotent. 

In particular det(l — TBg) = det(l — TA), therefore the set of matrices Bs satisfying det(l — 
TBs) = f{T) contains the set of all semisimple matrices which are S'0(A^, F£)-conjugate to A. 
Hence the lower bound 

|AI ^ {iBs,B^) I Bs 50(iV,F^)-conjugate toA,Bu e iCso(N){Bs))u(Fe)} , 

where Cso{N)iBs) (resp. {Cso{n){Bs))u) denotes the centralizer of Bg, seen as an algebraic 
group, under the action of SO(A^) (resp. the unipotent part of that centralizer). 

As already mentioned, we can not guarantee that the algebraic group Cso{N)iA) is connected 
and we will denote Cso{N)i^)^ its (connected) identity component. Then we argue as in |Chavl 
proof of Th. 3.5] to obtain 

KUnipotent elements of {CsoiN){A)f(Pi)}\ = e<^MCso(NM))°-MCsoiMM))^ . 

From [Bol II. 12.2, prop.], Cso{N){^)^ is a maximal torus in SO(A^) (indeed SO(A^) is a 
reductive group and from the separability assumption on /, we know A is regular semisimple), 
thus 

vk{CsoiN){A)f = vkSO{N) = ^. 
This finally yields the lower bound: 

I A I > .dimfC.nr «^ (A))0-N/2 \SO{N,Fi)\ 

' \Cso(nM){F,)\- 

Moreover, a Theorem of Steinberg asserts that the group of connected components of an 
algebraic group is always a subgroup of its fundamental group (see |SpSt[ II Cor. 4.4]). We 
deduce that Cso{N)iA) has at most 2 connected components. Adapting the result of Nori 
(see [No]) used by Chavdarov ( [Chavl page 160]), we obtain 

\CsoiNM){Fi)\ ^ 2(£+ l)^'-(^so(iV)(A))0 

Thanks to the formula (see, e.g., [Ar.El page 147]) on the cardinality of the special orthogonal 
group over F^ in the even dimensional case, we deduce 

(£_l)^(^-i)/2 ^ |50(Ar,F,)K£^(^-i)/2. 
Combining those last inequalities, we get 

|Aj »r^/2(£-i)^(^-i)/2, 

thus |A| ^ £N^/2-N ^ where, in these last two inequalities, the implied constant does not depend 
on N. □ 

The purpose of the last result we give in this section is to relate the cardinalities of a given 
conjugacy invariant subset of 0(A^, F^) and of the set of corresponding reversed characteristic 
polynomials in Mn^£. To that extent, its main interest lies in how we can apply it to our 
sieving problem. The arguments being very close to those used in the proofs of Theorem [15] and 
Proposition [TH it seems fair to include it in this "independent section". 

Lemma 16. Let N ^ 2 be an integer and £ be an odd prime number. Consider a subset @£ with 
cardinality On of Mjy^^^^i such that the elements / € 0^ satisfy 

(1) / is monic, 

(2) / is separable, 
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(3) disc(/) = disc(Q), 

(4) either /(—I) is a nonzero square 0/ for every f € @e, or /(—I) is a non square for 
every f £ &£. 

Moreover let ef'\e^P be two elements ofFi each equal to ±1 and such that the residue 

class in /F^ defined by condition (3) above and Lemma\M Let 

e, = {ge 0{N,F,) I (det,Nspin)(5) = (^^4'^ det(l - Tg)red € 6,} . 
Then, if 9£ = \@£\, we have 



t + 1 



At first, the above statement can look ambiguous as the integer Nred is not entirely defined 
by N but also depends on the matrix we consider. The point is, that once the determinant is 
fixed (which is the case in the Lemma since we restrict ourselves to matrices with determinant 

sf'^), there is only one integer N^ed that can correspond to N, so that, a posteriori, the assertion 
of the Lemma makes sense. 

Proof. Let us first consider the auxiliary set 

(5) {h e OiNred^F^) I (det,Nspin)(/i) = (l,e~f ),det(l - Th) = det(l - Th^d G Qi} , 

where is a fixed element of {l,eo} (which still denotes a set of representatives for F^/F^ ). 
We can trivially inject the set jS]) in Qg via the map h ^ h (B u where u is any representative 
of a fixed class of 0{N — Nredi Fg) modulo ^}{N — N^ed, F^) (that class corresponding to the 
couple (x,y) G {±1} such that {l,ef'^) x {x,y) = {e^f^\e^P)) and the quadratic structure on 
the corresponding (A^ — A^^ed)- dimensional being chosen with discriminant 1. Imbedding jS]) in 
Qi that way, we end up with a iVred-dimensional quadratic space having the same discriminant 
as the ambiant A^- dimensional quadratic space {V,Q). Moreover, from a fixed h in the set ^ 
we construct {'n{N,Fg) : Q{Nred,Fe)) distinct elements of &£. So, following the same idea as 
in [KoZetal Lemma 7.2], we can now compute a lower bound for Oi involving Oi. First, we have: 

^ ld?If'^F'^l E e 0(Ar,,,,,F,) I (det,Nspin)(5) = ),det(l - Tg) = f}\ , 

\i>l[I\red,i^ e)\ - 

/eef 

We know, thanks to the assumption (3), that Proposition [H] can be applied and so, that each 
summand of the right hand side of the above inequality is nonzero. More precisely each of these 
quantities is equal to the cardinality of the set A (depending on the polynomial /) of the proof 
of Theorem [T31 Following that proof and using the above inequality, we get 

\n{N,Fe)\ ^ e''f\SO{Nred,Fe)\ 



\mred,Fe)\ \CsoiN^^,){Af){F,)\' 

where, for each /€€>£, the matrix ^/ is the semisimple element whose existence is guaranted by 
Proposition [14] , Cso{Nrea)(^f) denotes the centralizer of Aj under the action of SO(A^red) and 
df is the dimension of the identity component of that algebraic group. The proof of Theorem [TSl 
yields 

|C50(7V..,)(^/)K 2(^+1)'^/. 
Now the derived group Q{Nred,Fg) has index 2 in SO{Nred,Fi) so we have 
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Thus 



1 



) 



1 



) 



A^red(Afred-l)/2 - 



i+1 



since we have df ^ dimS'O(A'red) = Nredi^red — l)/2- 



□ 



Remark. One can wonder why, in all the computations performed in this section, we always 
gave uniform bounds (with respect to the parameter A^) for the quantities studied rather than 
asymtotic estimates which, most likely, would have been easier to establish and would suffice for 
the argument needed in the proof of Theorems [Hand [171 This is because we have in mind another 
possible application (that we postpone to a future paper) involving L-functions of families of 
elliptic curves over function fields. In that work, we will show how the large sieve arguments used 
here can yield a lower bound for the proportion of elliptic curves with irreducible (up to trivial 
factors) L-function (seen as a Q-polynomial), when the curve varies in a suitable algebraic 
family, which is uniform with respect to the common conductor of the family (provided the 
related estimates for the local densities involved are uniform as well). 



In this last section, we state the main result of this paper which generalizes Theorem [T] 
in two different ways. To that purpose we show that for the two different kinds of group 
considered, Proposition [6] and Proposition [7] hold. Then, to derive our results from the large 
sieve inequality we need to find a suitable lower bound for the constant H. That issue, in 
the case where the groups involved are orthogonal groups, can be handled thanks to the results 
of Section [21 In the other case we consider (G = GL{n,A), = 5L(n, A)), the question of 
finding a lower bound for H turns out to be easier, as we do not have as many constraints on 
the matrices considered as in the former case. 

Let us first explain the additional terminology needed to state our result in full generality. 
Indeed, we need not (as Theorem [T] would suggest) restrict ourselves to the study of the ir- 
reducibility of characteristic polynomials of "random matrices", but the sieve setting we are 
working with enables us to study the Galois group of those Q-polynomials (see [KoSievel Chap. 
7] for the analogous study for the groups SL{n,Z) and Sp{2g,Z)). The Galois group of the 
(splitting field over Q of) the characteristic polynomial of a matrix of GL{n, Z[l/V]) can a 
priori be as big as the whole symmetric group 6^, but in the case of orthogonal matrices, ^ 
imposes conditions on the roots: if we denote N = n + m and N^ed = 2[(n + m)/2\ , the biggest 
subgroup of & N^^^ that can be realized as the Galois group of the characteristic polynomial of a 
matrix g € SO{n, m)(Z) is the group denoted Wjy^^^ which can be seen as the subgroup of &Nred 
permuting N^ed/'^ pairs of elements of {1,2,..., N^ed}- Of course if the Galois group of the char- 
acteristic polynomial of an element 5 G G is maximal (i.e. is equal to 6„ if G = GL{n, Z[l/V]) 
or to Wn^^j^ if G = SO{n,m){Z)) then the polynomial is irreducible. 

We can also state a generalized version of the second part of Theorem [H To do so, it is 
convenient to use (some of the basics of) the language of logic (as done in |KoDef| ). Recall 
(see Section 2 of loc. cit.) that a term in the language of rings is simply a polynomial / G 
Z[xi, . . . , Xn] and that an atomic formula f is a formula of the form f = g where / and g are 
polynomials (non necessarily in the same variables). Now if if is an atomic formula, ^ is a ring 
and if we assign the family of elements a = (oj) to the set of variables involved in the definition 
of ip, we say that ip{a) is satisfied in A and we denote 



if the equality which "is" ip is satisfied in A when the variables are given the values Cj. From 
atomic formulae we can build the so-called first order formulce by induction, using the symbols 



3. Statement and proof of the main result 



A ^ ip{a) , 
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-1, V, A and the quantifiers 3, V (we refer the reader to loc. cit. for examples of quite comphcated 
formulae that can be obtained in this way). Next, if ^{x) is a first order formula with respect 
to the variables x = (xi, . . . , and if A is a ring, then we denote 

ip{A) = {x e A"" \ ^{x)} . 

With such a terminology, we can state the main result of this paper: 

Theorem 17. With the above notation (as well as those used in the introduction) and assuming 
that the first condition of Proposition\^ holds and that n + m ^ 6 (resp. n ^ 2) in the case 
G = SO{n,m){Z) (resp. G = GL{n,Z[l/V]), we have 
(i) there exists a > such that for all k ^ 1, we have 

P(det(T — Xi:)red £ ^[^] reducible or does not have maximal Galois group) <C exp(— /33A;) , 

with (3s depending only on the underlying algebraic group G/Q, on the generating set S and 
on the sequence {ps)s (i-^- on the distribution of the ^k) o-nd where "maximality of the Galois 
group" means "with Galois group over Q equal to 6„ (resp. Wn^^^)" if G = GL{n, Z[l/V]) 
(resp. G = SO{n,m){Z) and N^ed = 2L(n + m)/2\). Moreover the implied constant depends 
only on G and the density of V in the set of all rational prime numbers (in the case where 
G = GL{n,Z[l/V])). 

(a) Let if be a first order formula in the language of rings with respect to the variables x = 
ixij)i<:ij^N (where N = n if G = GL{n,Z[l /V]) and N = n + mif G = SO{n,m){Z)). Set 

As = {i prime \ | (-(/^(F,)) n y,| • jG^r^ ^ 6} , 

and assume 

(6) there exists 6 > such that Ag has strictly positive Dirichlet density , 

then there exists a > such that for all k 1, we have 

P{A ^ ^{Xk)) « eM-Pik) , 

with the same dependency for jS^ as for (3^ and the same dependency for the implied constant as 
in the previous case. 

Remarks, (i) If G is the subgroup of integral points of a special orthogonal group, the fact that 
we emphasize (e.g. in Theorem the case where the quadratic structure is hyperbolic (i.e. the 
signature of the corresponding form is (n,n)) comes from the first condition of Proposition O 
Indeed, in the hyperbolic case, that condition is always fulfuilled (as we will see in Lemma [THl) . 
Another (somewhat artificial) way to ensure we can find in the general case a relation of odd 
length among the elements of a generating set S could be to add the identity element to S. 
Doing so we would end up with a "lazy" random walk for which we would have at each step a 
probability ph > to stay at the same point. 

The same problem occurs in the case where G = SL{2,Z[1/V]) (and this explains why 
this case is omitted in the statement of Theorem [1]); indeed, while the periodicity condition is 
always fulfilled for any generating system S of SL{n, Z[l/V]) if n ^ 3 (as will be proved in 
Lemma \W(ii) ). there are examples of such sets S for which that property does not hold in the 
case n = 2 (see [KoSievel Section 7.4] for further details). 

(a) The group denoted Wn^^^ that comes into play in the case of orthogonal groups has a 
more "functorial" description as the one given above. Indeed it is the so-called Weyl group of 
the algebraic group SO(A^). The fact that the Galois groups we investigate can be embedded in 
the Weyl group of the underlying algebraic group seems to be a quite general fact (it is proven 
by Kowalski for SL(n) and Sp{2g) in |KoSieve[ Chap. 7] and the case of (the split form of) the 
exceptional group E8 is treated in [JKZ] V 

Before getting into the details of the proof, let us give a few more remarks on part (ii) of the 
statement. First, for a fixed first order formula the set 

{^ip{Fe)) n Ye 
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(recall Yi = p^{a)G^i) in the above statement is in fact the sieving set 0^ with index I, in the 
notation of ^ and of the appendix. Next, to deduce the second part of Theorem [T] from {ii) of 
Theorem [171 we choose for (p the formula: 

ip{x) : \J 3y, = Xij . 

Thus, assuming Theorem [171 proving the second part of Theorem [1] is equivalent (once shown 
that the hypotheses of Proposition [7] are satisfied for the groups we study) to the fact that ([6|) 
holds for that choice of 

Finally, let us give examples of situations (i.e. choices of ip) for which jH]) holds/does not hold: 
consider for instance the case where (fiFg) is the set of F^-rational points of a subvariey V/Fi 
with codimension ^ 1 in G. Then ([6]) clearly holds since |F(F£)| is trivially bounded by (7£dimy 
where C is an absolute constant. In the opposite direction, if we investigate the probability with 
which the trace of a matrix in aG^ is sum of two squares, we quickly see that our method does 
not yield any quantitative information at all: indeed, if £ is an odd prime number, any element 
in F^ is the sum of two squares (a quite classical application of the pigeonhole principle) so 
that ^ is false for the choice 

N 

^{(xij)) : 3a, 3b, ^ Xi^i = 0^ + 6^. 

i=l 

The remaining sections are devoted to the proof of Theorem [T] and its generalization The- 
orem [T71 In order to handle the case of orthogonal groups, we first review some useful facts 
concerning certain quadratic modules over Z. 

3.1. Quadratic modules over Z. Let n,m be integers such that n + m ^ 4 and let {M,Q) 
be a quadratic module over Z with signature {n,m). The notion of spinor group we reviewed at 
the beginning of Section [2] can be extended to the case of quadratic modules (see |HMl Section 
7.2A]): in that more general case, we still have a morphism 

Spin(M) ^ 0(M) , 

the image of which is denoted 0(M) and the kernel of which is ±1 (see [HMI Th. 7.2.21]). In 
other words, these groups fit the exact sequence 

(7) 1 ^ {±1} ^ Spin(M) ^ n{M) 1 , 

where 0(M) can once more be seen as the simultaneous kernel in 0{M) of the determinant and 
the spinor norm (as defined in [HMI p 419]). In the arithemtic context we are interested in we 
will respectively denote by 0{n,m){Z), SO{n,m){Z) and n{n,m){Z) for 0{M), SO{M) and 
r2(M). Moreover, it will be convenient sometimes to see the first two of these groups as the 
groups of integral points of the algebraic groups 0(n, m)/Q and SO(n, m)/Q respectively. The 
properties we need r2(n,m)(Z) to verify, in order to apply Propositions [6] and [7] are contained 
in the following lemma 

Lemma 18. (i) For integers n, m such that n + m ^ 4, we have 

(1) If d ^ 1 is a squarefree integer whose only prime factors are outside a fixed finite set S, 
then the reduction modulo d: VL{n,m){Z) il{n,m){Z/dZ) is onto. 

(2) Property (r) holds for Q,{n,m){Z) with respect to the family of its congruence subgroups 
(ker(pd : 0(n,m)(Z) ^ J7(n, m)(Z/dZ)))^^^ . 

{a) If in addition the quadratic module considered is hyperbolic (in the sense of the introduction, 
in particular n = m), and if 3, we have 

(1) 0(n,n)(Z) is finitely generated. 

(2) For every symmetric generating system S of i}{n,n){Z) there exits a relation of odd 
length c inside S: 

Sl ■ ■ ■ Sc = 1 , Si & S . 
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Proof. For (ii) we use the useful elements of the automorphism group of 0(n, n)(Z) called Eichler 
transformations (see |HMl 5.2.9]). As the ambiant quadratic module {M,Q) we consider here 
is hyperbolic, we know from Theorem 9.2.14 of loc. cit. that these transformations span the 
subgroup 0(n,n)(Z). This proves Moreover the same result asserts that the unitary 

elementary transformations Eij{l), where i ^ j run over a finite set of indices, span the group 
r2(n,n)(Z) and satisfy the commutator relation 

[Eij{l) : Ek,iil)] = Ei^lil) , 

if i,j,k,l are distinct and run over the same (suitably chosen) set of indices (see |HMl Th. 
9.2.14]). Hence 0(n,n)(Z) equals its own derived group. Now to prove {ii){2) let us assume, 
by contradiction, that there is no relation of odd length among a fixed symmetric generating 
system S of r2(n,n)(Z) and let F{S) denote thre free group generated by S. The morphism 

FiS) {±1} 

S I— 5- — 1 , 

induces a surjective morphism 0(n,n)(Z) {±1}. Thus a quotient of ri(n,n)(Z) is isomorphic 
to {±1}; this contradicts the fact that r2(n,n)(Z) is its own derived group. 

For (i)(l) we first use strong approximation to justify the surjectivity of the reduction 

TTp : Spin(n,m)(Z) — > Spin(n, m)(Fp) , 

where p runs over the set of all prime numbers which do not lie in a fixed finite set S. Indeed, 
using the fact that the group Spin(n,m)(R) is not compact (the algebraic group Spin(n,n) 
is said to be of non compact type), we can apply Borel's Density Theorem (see jPRi Th. 
4.10]). Now Spin(n, m)(Z) is a Zariski dense subgroup of the simply connected algebraic group 
Spin(n, m)/Q, so strong approximation can be applied; more precisely, thanks to fPRl Th. 
7.15], we deduce the surjectivity of iTp provided p remains outside of a finite set S of prime 
numbers. 

Then, using ([7]), we can consider the diagram with exact rows 

1 > {±1} > Spin(n,m)(Z) > n{n,m){Z) > 1 

TTp 

1 > {±1} > Spin(n,m)(Fp) > Spin(n, m)(Fp)/{±l} > 1. 

If p is a prime not lying in S, the two left vertical arrows of the above diagram are onto (as we 
have assumed to be working only with odd prime numbers). Then, if we define the last vertical 
arrow in such a way that the diagram commutes, this map must be onto as well. Moreover, it 
is easily seen that this last arrow also corresponds to the usual reduction modulo p 

n{n,m){Z) Q{n,m){Fp) . 

Now, for a squarefree integer d without any prime factor in 5, we invoke Goursat-Ribet's 
Lemma (as stated in [Chavl Prop. 5.1]). Indeed, for p ^ S, the group 0(n,m)(Fp) has no non 
central proper normal subgroup and the group Q,{n,m){Fp) modulo its center is a simple group. 
So, for such a d, we have a surjective morphism 

J7(n,m)(Z) ^ n{n,m){Z/dZ) . 

Finally, for (3), we can apply [HVl Th. 8, page 23], thanks to which we know that, as 
n + m ^ 4, the group SO{n,m){'R) has Kazhdan's Property (T). Combining this with |HVl 
Cor. 5 to Th. 4 page 33] we deduce first that SO{n,m){Z) has Property (T) and then that 
0(n,m)(Z) also has Property (T). The weaker Property (r) fot r2(n,m)(Z) with respect to the 
family of its congruence subgroups follows immediately. □ 

Remark. In the above proof, we use the notation Spin(n, m)(Fp) or J7(n,m)(Fp) just to keep 
track of the indefinite quadratic form over Q giving rise to the matrix groups for which we then 
take the reduction modulo p. 
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3.2. Proof of the main Theorem. Recall that we are working out the two following cases: 

• Either, for n + m ^ 6 and with notation as above, G = ^(^(n, m)(Z), = r2(n,m)(Z), 
and r = S'0(n, m)(Z)/0(n, m)(Z), in which case, these groups fit the following commutative 
diagram with exact rows and surjective left vertical map 

1 > 0(n,m)(Z) > SO{n,m){Z) > T > 1 



1 > Q{n,m){Fp) > SO{n,m){Fp) > Tp > 1, 

provided p ^ S (see Lemma [T8]l . and where Tp denotes the abelianization of SO{n,m){Fp). 

• Or, for n ^ 2 and with notation as in the introduction, we set G = GL{n, Z[l/V]), G^ = 
SL{n, Z[l/V]), r = Z[l/V]^ , in which case these groups fit the following commutative diagram 
with exact rows 

1 > SL{n,Z[l/V]) > GL{n,Z[l/V]) > T > 1 



1 > SL{n,Fp) > GL{n,Fp) > F^ 



provided p ^ V. The question of the surjectivity of the downward arrows Hp (which still de- 
note reduction modulo p) is easily answered here. Indeed, it is straightforward to check that 
GL{n,Z) — > GL{n,Fp) is surjective and the fact that its restriction SL{n,Z) — > SL{n,Fp) is 
also surjective is well known (see, for instance |Shl Lemma 1.38] where the surjectivity is proved 
in the more general case of the reduction SL{n, Z) — > SL{n, Z/dZ) modulo any positive integer 
d). As we want to apply Propositions [HE] and [7] to this case, we need the following analogue for 
the points of Lemma [l8] 

Lemma 19. (i) For n ^ 2, we have the following properties: 

(1) SL{n, Z[l/V]) surjects onto SL{n,Z/dZ) for each squarefree d without prime factors in 
V. 

(2) SL{n, Z[l/V]) has Property (r) with respect to the family of its congruence subgroups 



kei{pd : SL{n, Z[l/V]) SL{n, Z/dZ))^ ^ , 

where d runs over the set of squarefree integers without prime factors in V. 
(a) If we suppose n ^ 3 then, for every symmetric generating system S of SL{n,Z[l/V]), 
there exists a relation of odd length c: 

si - ■ ■ Sc = I, Si e S . 

Proof. We have just discussed point (i)(l) and (i)(2) is proven in Lemma lU 

For (ii), we note that, as the ring Z[1/'P] is euclidian, the group SL{n, Z[l/V]) is generated 
by the (infinite) set of transvection matrices Tij(a) (the sum of the identity matrix and the 
matrix with all entries equal to zero except for the one in position which equals a), where 
a € Z[l/P]^ and 1 ^ i ^ j ^ n. Such matrices satisfy the commutator relation 

[Tijiai),Tj^k{a2)] = 7i,fc(aia2) , 

as soon as k are pairwise distinct (such a choice is indeed possible since n ^ 3). Prom that 
equality, we deduce that SL{n, Z[l/V]) equals its own commutator subgroup. The end of the 
proof is then exactly the same as for the last point of Lemma [18] □ 

We are now ready to prove Theorem [T7] Thanks to Lemmas [18] and \19\ Propositions [6] 
and [7] hold with data corresponding to the two cases described above. To prove the exponential 
decrease of the probabilities investigated as k grows, we need to give a suitable lower bound for 
the constant 
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where C is the set of primes in A up to a fixed L ^ 1 and A is a set of primes with strictly 
positive Dirichlet density that we will make precise in due course. 
Both for the conjugacy and the non-conjugacy coset sieve, we have 



We shall prove that 

le 



»1, 



with an implied constant depending only on n and on the underlying algebraic group G, for our 
different choices of sieving sets 0^ and groups G\. Such an estimate will turn out to be sufficient 
to prove Theorem [TT] (and deduce Theorem 

First we note this inequality is quite obvious in the setting of (ii) of Theorem [TT] with the 
choice A = A5 (the hypotheses are chosen puposely for that estimate to be true). Indeed, in 
that case, we have 0£ = {^Lp{¥i)) n 1^, so that, by assumption, there exists a (5 > such that 



\a\\ 

for all £ € A5. Then, looking back at ([2]) and using Proposition [7] as well as the Prime Number 
Theorem, we can easily get (ii). Indeed, we have 

P(F^ \= ip{pe{Xk)) for all i ^ L) ^ {I + 1^^'''^+^)/^ exp{-T]k))L-^ logL , 

with an implied constant depending only on n and the density of A^ as a subset of the rational 
primes. Setting L = ex.p{jj^^), we obtain {ii) of Theorem [T71 

Now as far as (i) of Theorem [TT] is concerned, the set A we choose is, depending on the 
type of group considered, either the set of primes which are not in S (see Lemma [T8l) or the 
complementary of P in a subset (with finite complement as we will see later) of all prime numbers 
(see Lemma fTOl) . The conjugacy coset sieve enables us to study the reduced characteristic 
polynomial of as k grows. For a fixed a £ G, we choose: 

(9) ei = {ge peia)Gj \ det(r - g)red G , 

where Qi is, for each I, a set of polynomials having imposed factorisation patterns. Each 6^ 
will be a conjugacy invariant subset of Yg. Moreover, in the case where G = 50(n, m)(Z) and 
G^ = r2(n, m)(Z), the set can be seen as a subset of 0{n + m,F() consisting of matrices with 
fixed determinant (equal to 1) and fixed spinor norm. 

Whereas the estimate ([8|) will be derived directly, for suitable families (B^) in the case where 
G = GL(n, Z[l/P]), from |KoSieve[ Appendix B], the analogue study for G = SO{n,m)(Z) 
requires a few additional computations based on the ideas of |KoZetal Lemmas 7.1 and 7.2]. 
The strategy of loc. cit. relies on the fact that as soon as a few particular conjugacy classes of 
Wn^^^ (where we keep the notation Nj-ed = 2[(n + m)/2j) are detected in the Galois group of a 
polynomial P G Z[T], the Galois group of P is necessarily isomorphic to the whole group W]\f^^^. 
It is well-known that such conjugacy classes can be detected through the study of the factorisa- 
tion patterns of P (modi) with i taking many different prime values. As explained in jKoSievel 
Lemmas 7.1 and 7.3], it is enough to consider four distinct families (G^) or equivalently (via J!])) 
four families of sets of polynomials (G^) (for simplicity we will denote in the sequel N = n + m 
and Nred = 2 [(n + m) /2j ) : 

(1) Let Q^^'* be the set of polynomials / in Mtv^^^,^ 

• which are irreducible if N is odd or which are irreducible with a fixed value mod- 
ulo nonzero squares of in —1 and satisfy disc(/) = disc(Q) if N is even and 
0{Nred,Fe) = 0{N,Fi) is nonsplit, 
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• which factor as a product of two distinct monic irreducible polynomials of degree 
Nred/'^ if ^ = ^red is even, 0{Nred,Fe) is split and 1=1 (mod 4), 

• which factor as a product of an irreducible monic quadratic polynomial and an 
irreducible polynomial of degree A^^ed ~ 2 if N = N^ed is even, 0{Nred^ ^e) is split 
and i = 3 (mod 4). 

~ (2) 

(2) Let 0) be the set of polynomials / in M^^^^^i with a fixed value modulo nonzero squares 
of F£ in —1, which satisfy disc(/) = disc((5) and which factor as a product of a monic 
quadratic polynomial with distinct monic irreducible polynomials of odd degrees. 

~ (3) 

(3) Let 0^ be the set of polynomials / in M^^^^^i with a fixed value modulo nonzero squares 
of in —1, which satisfy disc(/) = disc((5) and with associated polynomial h (such that 
/ = x"^h{x + x~^)) being separable with at least one factor of prime degree > N^ed/^- 

(4) Let 0^^^ be the set of polynomials / in Mj^^^^^i with a fixed value modulo nonzero squares 
of F£ in —1, which satisfy disc(/) = disc(Q) and with associated polynomial h being 
separable with one irreducible quadratic factor and no other irreducible factor of even 
degree. 

Lemma 20. For 1 ^ i ^ 4 we have, with the above notation. 



with an implied constant depending only on N . 

Proof. For (G^^'*)^, let us first consider the case where N = N^ed is even. It is enough in the 
nonsplit case to combine Lemma [T2l and Lemma [T6l This yields: 

\n{N,Fe)\ ^ 2N\ e )\ e+i) 

from which we derive the estimate we want. 

In the split case, the estimate we need is exactly the statement of the lemmas 6.5 and 6.6 
of |KaL| . Indeed, we deduce directly from loc. cit. 

ie(^)| _ 1 



\n{N,Fe)\ ^ 4Ar2 ■ 

Now if N is odd, we invoke a new lemma of [KaLl Section 6] (namely Lemma 6.4 of loc. cit.) 
from which we get: 

ie«l ^ 1 



\n{N,Fi)\ 2N-2' 
for£^max(7, {N-l)/2). 

As far as (G^ )^ is concerned, we argue as in the proof of Lemma [16] (when we embedded ^ 
in G^) so that we can work with a quadratic space Vred of dimension Nred having the same 
discriminant as the ambiant A'^-dimenional space. Then, notice that we can take as a model of 
our quadratic space 

Vred = (Fj, nonsplit) (F^'''"'''^, split) , 
if 0{NrediFi) is the nonsplit model for the orthogonal group in dimension Nred over F^, and 

Vred = (Ff, nonsplit) © {F^'''"'^~'^ , nonsplit) , 

if we deal with the split model for the orthogonal group. 

Now we proceed as in the proof of [KoSievei Lemma 7.3] i.e. we consider separately the case 
where Nred/^ is even and the case where Nredf^ is odd. We perform the same computation 
as in loc. cit. with the slight difference that the (only) irreducible quadratic factor of each of 
the polynomial we consider takes imposed values modulo squares in 1 and —1, so that we need 
to invoke Lemma [12] to deduce the number of possible quadratic factors is greater or equal to 
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(£/4) X (1 - 6/^) (instead of {l/2){l - I /I) in tKoSievei Proof of Lemma 7.3(ni)]). So combining 
loc. cit. and Lemma [TBI we get 

lel^^l 1 / lxAf,,d/2-l/ In/ 1 \NrUNr,d-l)/-i 



Q.{2n,¥i,)\ ANred^ i) \ U)\ £ + 1 

1 / InA^/2-1 / In/ 1 N Af(JV-l)/2 



hence the estimate we want to prove 

Finally, for (of and (of ^)^, it is 
and Lemma [16] yields the estimate of Lemma [20) □ 



Finally, for (G^^^)^ and (B^^^)^, it is straightforward to verify that the combinaton of Lemma[T3] 



Next we turn to the case of the conjugacy coset sieve for aSL{n,Z[l/V]). In that case, we 
have, for each i ^V, Gf = SL{n,Fi) and the sieving sets we choose are still given by ^ with 
this time, for any conjugacy class c G 6„ whose elements have a decomposition in disjoint cycles 
involving rii cycles of length i for 1 ^ i ^ r , 

0£,c = {/ ^ I / has factorisation type c and /(O) = det(p^(a))} , 

where we say that a monic separable polynomial / G Fi[T] of degree r ^ 1 has factorisation 
type c £ Sr if / factors as 

/ = /i • • • /r , 

where is a product of dictinct irreducible monic polynomials of degree i and ^ irii = r. 

In the case of the trivial left coset pi{a) = 1, the estimate we need is given by Kowalski 
in [KoSieve[ Appendix B, Lemmas B.2 and B.5]. For the case of the general left coset, it is 
straightforward (by performing the obvious change of variable sending pe{a) to 1) to verify that 
the same estimate holds, so that we get the following result: 

Lemma 21. With the above notation, we have 

\Qe,c\ 
\SL{n,Fe)\ 

as soon as i > IGn? , with an implied constant depending only on n. 

Note that, to perform our sieve, the above statement suggests we should remove the primes 
smaller than 16n^ from A but this does not affect the final result as the Dirichlet density of A 
remains unchanged. 

Now we use the inequality: 
P(det(r — Xk)reddoes not have maximal Galois group) ^ ^p(Gai(det(r-Xfc)) ne* = 0) , 

where is the conjugacy class of Wn^^^ (resp. ©„) determined by the family 6 = {@e)e 
and where the sum runs over the family (0^*^)i^i^4 (resp. (0c)cg@tt ) if G = 50(n, m)(Z) and 
Nred = 2[{n + m) /2\ {resp. G = GL{n,Z[l/V]). 

Looking back once more at ^ and applying both Proposition [6| and the Prime Number 
Theorem, we obtain, for the two types of groups investigated 

P(det(r - Xfc)reddoes not have maximal Galois group) < (1 + L^^'^+^^Z^) exp{-r]k)L~^ logL , 

with an implied constant depending on n and the (strictly positive) density of A in the set of all 
rational primes. If we set L = exp(^^), then choosing for (3s any positive real number smaller 
than yields (i) of Theorem [171 
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3.3. Proof of Theorem [H As a conclusion, we explain how to deduce Theorem [T] from Theo- 
rem [171 Notice first that the first part of Theorem [1] is a trivial consequence of (i) of Theorem [T7l 
As explained in the previous subsection, the only thing we need to prove to get the second in- 
equality of Theorem [T] is that the first order formula 

ip{x) : y 3y, = Xij , 
yields sets = (-i99(F£)) n Yi (indexed by a set of primes A^ to be determined) such that 

|e,|-|Gfi-i»i. 

For both the case where G = SO{n,m){Z) and G = SL{n, Z[l/V]), the above sets &£ can be 
expressed in the following way 

&£ = {g = i9i,j) e a^Gf I gij is not a quare in A/ (i) = F^} , 

with notation as in Theorem [T] and where denotes Q{n,m){Fi) or 5-L(n,F^) depending on 
what is G. The element ag = pg{a) determines, in both cases, the left coset of Gg (with respect 
to G^) which contains Qg. 

We first consider the case where G = SL{n, Z[l/V]) (so Gf = SL{n,Fi) for i V). As a 
representative of the left coset of GL{n,Fi) consisting of the matrices with fixed determinant 
say di, we can choose the diagonal matrix ai given by i) = di and = 1 if i ^ 2. Using 

the Legendre character (j) to detect squares, we want to evaluate 



2\SL{n,Fe)\ ^ fj^^ ^ ^ W 



Thus to obtain the inequality |0f | |5'L(n, F^)| ^ ^ 1, it is enough to prove 



E 

g&{atSL{n,¥i))i^j 



where we denote {aiSL{n,F the matrices of aiSL{n,Fi) with a nonzero entry in position 
(i,j), and where d equals v? — 1, the dimension of the algebraic group SL(n). 

Now, for each g G agSL{n^F i), there exists h G SL{n,F() such that g = agh. The (i,j)-th 
entry of g is given by 

9i,j = '^Oii^i^i^k)hkj = ai^i^i^i)hij , 

k 

since the matrix ag is diagonal. So it is enough to prove 

(^) E (^)«^'^-^/^ 

heSL{n,Fe)i^j 

which can also be written in the following way: 



heSL{n,Fi)ij 

This inequality is proved in |KoSieve[ Appendix B., Prop. BA]. Thus ([6]) of Theorem [T71 (ii) 
holds if we choose for A^ the complementary of V in the rational primes, and we deduce the 
second part of Theorem [T] in the case where G = GL{n, Z[l/V]). 

Finally, in the case where G = SO{n,m){Z) (i.e. Gf = fl{n,m){Fe)), things are slightly 
different as this time, the group 0(n,m)(F^) cannot be seen as the group of F^-points of an 
algebraic group. In order to end up applying the same techniques as above, we need to relate 
for fixed indices 1 ^ i,j ^ n + m the cardinality of the sieving set 

&£ = {g & a£Q,{n,m){Fi) | gij is a square inF^} , 
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to the cardinality of 

0f*^ = {5 £ SO{n,m){Fi) \ gijis a square inF^}. 

To that purpose, we first make a special choice for the basis thanks to which we identify the 
elements of SO{n, m){Fi) with their matrix representation. Indeed the surjectivity of the spinor 
norm (see Section[2]) onto {±1} enables us to choose a vector, say ei, such that Nspin(?'ei) = —1, 
where denotes the reflection with respect to the hyperplane Tie^ which is orthogonal to ei. 
We can consider the restriction of the quadratic form Q on the ambiant space to Tiei- This 
restriction is still non degenerate ( [OMl page 139]) and we can choose the second vector 62 of 

the basis we are constructing in such a way that the corresponding reflection rea*"^ of Tiei has 
spinor norm 1 (see [KaLj Proof of Lemma 6.3]). Then we can complete the basis of Wei with 

vectors {es, . . . , e„+m} in such a way that the matrix of rej"^ written in the basis (e2, . . . , en+m) 
has diagonal coefficients (—1, 1, . . . , 1) and zeros outside the diagonal. Finally we can extend 
to the whole ambiant space by making it act trivially on ei. We get a reflexion of the whole 
space. Now the product re^rej is an element of SO{n,m)(Fi) with spinor norm —1 (note that 

Nspm(?"e2) = Nspin(r'e2''^ )) and the matrix Me^^e2 oire^r^^ in the basis {ei)i is the diagonal matrix 
with Me^,e2(^)^) = — 1 if i = 1,2 and Me^^e2ih'i) = 1 otherwise. Now consider the involution 

where e is any representative of the left coset of SO{n, m){F£) consisting of elements with spinor 
norm —1 and 

= {g £ (ea£)J7(n, m)(F^) | gij is a square inF^} . 

We have {Me^f.29)i,j = ~9i,j if J = 1,2 and {Me^e29)i,j = 9i,j otherwise. So, for primes 
i = 1 (mod 4), gij is a square in F^ if and only if {Me^e29)i,j is a square as well. For such primes, 
we deduce that, 

lef^l = le^i + |e|| = 219^1. 

So 

2\n(n,m)(Fe)\ ^ \ min.mMF t)\ ^ \ \l 

g(iaiQ.{n,m)(^t.) gG50(n,m){Ff ) 



E (^- 



2|SC>(„,m)(r,)| 

g<^SO(n,m)i^i) 

9i, jT^O 



as soon as we restrict ourselves to primes such that £ = 1 (mod 4). 

To deduce from the last inequality that |0£||O(n,m)(F^)|~^ ^ 1 (with an implied constant 
depending only on n and m), we use the exact same argument as in the previous case (where 
the finite group involved was SL{n,Fi)). Indeed, we can now see the sum investigated as a 
character sum over the F^-points of the geometrically irreducible variety SO(n, m)/F£. 

So we can choose A5 = {primes congruent to 1 modulo 4} (which has Dirichlet density 1/2) 
and then apply once more (ii) of Theorem [TT] to get the second part of Theorem [T] in the case 
G = SOin,m){Z). 
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4. Appendix: Coset Sieves 



The purpose of this appendix is to explain the role that the large sieve plays in the proof 
of Theorem [T71 We give here the full statements with proofs of the different kinds of a priori 
estimates we need to get the kind of explicit upper bounds of Theorem [T71 The results we expose 
here are very much in the spirit of [KoSievej (especially Section 3.3 of loc. cit.) and sometimes, 
we only recall some results of [KoSievej . Moreover in the last subsection, we give self-contained 
versions of these statements in order to make it possible for the reader to follow the proof of 
Theorem [TT] without having to get too much involved in the details of the sieving machinery. 

4.1. The general framework. Our general sieving context is that of the coset sieve. A general 
description of that sieve goes as follows: we suppose we are given a discrete group G with a 
normal subgroup such that the quotient F = G/G^ is abelian. Moreover, we suppose that 
there exists a subset A of the rational primes such that, for any ^ G A, we have a surjective 
group homomorphism 

Pi : G ^ Ge , 

where Gi is a finite group. 

That is, of course, a very natural generalisation of the reduction modulo £ morphism from Z 
to Z/iZ. We emphasize here the fact that the above data is really all we need to set the sieve 
for cosets that we apply (and this is really the strong idea underlying [KoSievei Chap. 3.3]). All 
the framework we build from there, comes from "natural" deductions. First, we denote 

= PiiGS) 

for i £ A. That subgroup is normal in Gi because G^ is a normal subgroup of G and, since 
for every i £ A, the morphism pi is onto. Now, all these groups fit the following commutative 
diagram with exact rows (such a diagram can already be found, in a geometric context, in |Chavl 
Th. 4.1], and is extensively used in [KoZeta] ): 

1 > G9 > G — ^ F = G/G3 > 1 

(10) 

1 > Gf > Ge F, = Ge/G'i > 1 

where the surjective morphism pr^ is defined in such a way that the diagram commutes. In 
both rows, the quotient map is denoted d, in order to avoid the introduction of an additional 
notation. 

An important feature of that sieve setting, in view of the proof of (z) of Theorem [T71 is that 
each left coset aG^ of G {a being any fixed element of G) is conjugacy invariant. This comes 
from the fact that the quotient F = G/G^ is an abelian group. 

We now fix an element a € G, and, use, from now on, the upper index " to denote the set of 
conjugacy classes of the (conjugacy invariant) set considered. The two following sieve settings 
(y, A, {p£ : Y ^ Yg)) will be useful for our purpose: 

• either Y = (aG^)^, = {pe{a)G^)'^ and pe also denotes the restriction (which remains 
surjective) Y ^ Yg, for any £ € A (this will be referred to as the conjugacy coset sieve). 

• or y = aG^, Yi = pi{a)G^f and pi also denotes the (surjective) restriction Y —i- Y^, for 
any £ £ A (this will be referred to as the non-conjugacy coset sieve). 

As we do in the introduction, we assume we are given a probability space (^',E,P). The 
random walk (Xk) we are interested in (see the introduction again), can be seen as an application 
on ^' with values in Y (whatever choice we make for the set Y among the two possibilities above). 
For each k, we end up with a siftable set (^', Afc,P). Following Kowalski's book [KoSievej . let us 
denote by C* (the prime sieve support) the set of elements £ £ A that are smaller than a fixed 
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Pi 



integer L ^ 1. The large sieve method we use here consists in giving, for any family = (B^) 
of subsets of Yi (the sieving sets) indexed by A, an upper bound for the probability 

P({x E ^ I PiiXkix)) for alie G £*}) . 

For a matter of convenience, we will rewrite this probability in the standard way 

P(pK^fc) 0£ for ain G £*) . 

We need to make precise the meaning and the definition of the constants A and H appearing 
in the fundamental inequality [2] (the constant A is sometimes denoted A{Xk,C*) when we want 
to emphasize the dependency on the parameters). Prom now on, we will assume we are given a 
probability density U( on Q( for any ^ G A; then the constant H can be taken to be equal to 



To define the large sieve constant A(X/c,£*), we should first emphasize that the central issue, 
in order to get a useful upper bound from jj]), is to find a suitable basis for the space L'^iY^, U() 
which is the complex Hilbert space with associated inner product (defined for C-valued functions 
/ and g on Y^): 

= X] My)f{y)9{y) ■ 

If denotes an orthonormal basis of that space containing the constant function 1 and if 
Bl = B(, \ {1}, then, for any square integrable function /? : — > C, the large sieve constant 
A(Xfc,£*) is defined as the smallest constant A satisfying 

E E I / /?M¥'(pK^fc(^)))rfPM ' ^ A / \[3{u;)\^dV{u), 
which can also be written, denoting by E(X) the expectation of a random variable X, 

e&c* <f£B} 

The proof of the inequality ^ as we state it, can be found in [KoSievei Prop. 2.3]. To find an 
upper bound for A{Xk, C*), we use (see |KoZetal Section 5] for an analogue given in a geometric 
context): 

(11) A < max max J] ^ \W{ip,ip')\ , 

where the W{(p, ip') are the "exponential sums" given by 



(12) W{^,ip') = E(v9(p,.(Xfc))v?'(p,(Xfe))). 

Obviously the usefulness of ([TT]) lies in the fact that it now suffices to give estimates for the 
individual sums W{(p^ ip') to deduce an upper bound for the large sieve constant. In our context 
where the sets Yi are left cosets in finite groups, it is natural to use the irreducible characters 
of the finite groups in order to construct a suitable basis B^. Moreover, that explains why it 
seems fair to call the W{ip, ip') exponential sums. 

4.2. Exhibiting orthonormal bases. In what follows, we give the description, for each ^ G A, 
of the basis Bi we need (note that we need to handle both the conjugacy coset sieve and the 
non-conjugacy coset sieve). 

First, we recall the following Lemma, due to Kowalski (see jKoSievel Lemma 3.2]) in which a 
basis for L'^{Yi, v^) is described in the case of a conjugacy coset sieve (we recall that in that case, 
Yi = {pi{a)G^f^^) where the density vi is the uniform density defined by viiy^) = jy^HG^I"^. 
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Lemma 22. With the same notation as above, let 

:y«GG«^TrKy«)), 

where ^ € A and vr is an irreducible representation of Gf . Let Lp'iY^^ U() denote the Hilbert space 
of complex valued square integrable functions with respect to the scalar product: 

Then we have 

(1) If TT and T are irreducible representations of Gi, 

_ 0> 2/ ^\Gl 9^ T\Gl or ip^\Y, = 0, 

(13) {^7T;y^T) 



'il^{d o p^(a))\TJ\ , otherwise , 

where is the character group of Ti and ip is an element of the group = {^/^ G | 

vr ~ vr (8) V'}- 
(2) Let Bi be the family of functions 

where vr runs over the subset FE^ of a set of representatives for the irreducible repre- 
sentations of Gi with respect to the equivalence relation 

IT 1^ T if and only if ir^cf — T\Gf i 

and where vr € n| if and only if f-K\Y^ 0- Then the family Bi is an orthonormal basis 
for L^Ye,iye). 

In the case of the non-conjugacy coset sieve, the irreducible representations of Gi cannot be 
used in such a direct way to construct Be, but in spite of that, they turn out to be very useful 
once more. 

Fix an £ S A and a finite dimensional irreducible representation vr^ of Gi. Let 

B = (e^ gdimTrn 

denote an orthonormal basis of the representation space K-^ of vr^, with respect to a G^-invariant 
inner product on denoted (; (note that, pi having finite image, it is always possible to 
assume the existence of such a G^-invariant inner product). Then for any two elements e and / 
of {e^^, ■ ■ ■ , e^™'^^}, consider the function 



TTi,e 



J : X £Gii-^ \/dimvr£(7r^(j;)e; f)-^^ , 



called a matrix coefficient. 

Then the family (v^jr^^e,/) obtained by varying vr^ in 11^ (where 11^ denotes a set of respresen- 
tatives for the isomorphism classes of irreducible representations of Ge) and e, / in a fixed basis 
S^^, forms an orthonormal basis for L'^{Gi, ug) of square integrable complex valued functions on 
Ge with respect to the inner product 



corresponding to the uniform density ue defined for y € Ge by veiv) = (see [Knl Section 

L5] for a proof). 

From that result we can derive, in the non-conjugacy sieve setting, a useful orthonormal basis 
for L'^{Ye,i'i), where we recall that Ye = pe{a)Gf and where, for ?/ G Y^, ueiy) = \Gf\~^: 
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Lemma 23. With notation as above, consider the inner product on Lp'iY^^u^) defined by 

LetiTg andr^ be irreducible representations ofG^ and (e, /) (resp. {E,(j))) a couple of elements 
of an orthonormal basis B^^^ (resp. B-,-^) of (resp. Vr^). The functions ^TTi,e,f o-nd '^Ti,e,4> 
are said to be equivalent (in which case we will note v^tt^.c/ ~ ^T(,e,4>) if the entry (e, /) of 
MatB„ TT£{g) and the entry {e^cj)) of Mais^ Tiid) coincide for all g G G^. 

Then 

(1) If TT^ and Ti are irreductible representations of G^, and if we denote 

f r""' = {X e I ip^,,e,f ® X = ^n„e,f inL^Ge)} , 

we have 

if(Pne,e,f 7^ ^Ti,e,<lyOr if the entry {e,f) (resp.{e,(f>)) 
of Matfi^ 7r£(5) (resp. MatBrJiig)) is zero for all g eYe, 

^ V'(d(a£))|f J''^'"'-' I otherwise, whereag = pe{a), G f ^ andipj^^^ej V' - ^n,s,4 

(2) Let Bi be the family of functions 

Y,^ C 

where (7r^,e,/) runs over the triples corresponding to a system of representatives for the 
equivalence relation ~ and where we assume that for every triple (7r^,e, /), there exists 
an element g E Yi such that the entry (e, /) of MatB^^^T^iig) is nonzero. Then B(, is an 
orthonormal basis for Lp'{Yi, ve). 

Proof. (1) We evaluate the scalar product 



d{y)=d{ai) 



Yl {Y '^i(^i)i'iy))'Pne,e,fiy)vre,eAy) ^ 



where the last inequality is obtained using Probenius reciprocity. 

Obviously the right hand side of the above equality vanishes as soon as </^7r£,e,/|y^ or ^PTi,e,(l>\Yi 
identically zero (which corresponds respectively to the vanishing of the entry (e, /) of MatB^^7r£(5) 
or (e, 0) of MatB^^r£(5), for all g G Y^). However, if that quantity does not vanish, we have, on 
the right hand side. 

Now, in L^(l£,f£), we have the equality of functions ^-n^^ej ® '4' = V-wi^^.eJ- Indeed any 
G^-invariant scalar product ( ; )7r^ on T^-^ — V-ki®iI) (as vector spaces) remains G^-invariant if G^ 
acts via tt^ (8) V' (which is still an irreducible representation of G(). We deduce 

where 6 denotes Kronecker's symbol. 
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The quantity 6{(p^^,^^^e,f, fTi,e,<j)) equals 1 if and only if (/7^^,g,^,e,/ = fTi,e,<t> in L'^{Gf). This 
condition is equivalent to the coincidence of the restrictions ipT^^^^j^Qa and '^Ti,£,(i>\QS , i-e. the 

equality between the entry (e, /) of MatB^^7r£(g') and the entry (e,0) of MatB^^r£((^), for all 

g £ G^. In that case we deduce, using |KoSieve[ Lemma 3.2], 

where ip is any of the characters of such that 

fTTe,eJ V' - Vn,e,(P ■ 

The assertion (2) is straightforward using (1) and the above arguments. 

□ 

Remarks, (i) In the course of the proof of Lemma [231 we have seen that the relation ^■Ki,ej ~ 
fTi,£,4, is equivalent to the existence of a character ip €Ti such that 

in L?{G£). Then we proved that the scalar product 

is equal to ilj{d{a())\{x G L^ | ^■Ki,e,f ® X = ^iTi.ej in L^(G^)}[. Thus, if vr^ is an irreducible 
representation of Gg, the group f^^, in the sense of Lemma [22l is a subgroup of f J"*^'^'^ for any 
choice of vectors e, / in an orthonormal basis of the representation space T^^. Indeed, if -0 S T^, 
we have an isomorphism of G^-representations: tt^ (8) V — T^t, hence, 

for every g £ Gg. That means that the equivalence relation of Lemma [22] is "stronger" than 
the one described in Lemma [23] (in the sense that the classes for the former relation, which are 
contained in those for the latter, may form strict subsets in those classes). 

(ii) Using the example of the dihedral group -D„, neven ^ 2, we see that the equivalence 
relation ~ defined in Lemma [23] can actually be non trivial, i.e., we can exhibit two non isomor- 
phic irreductible representations (vr,!^-) and (t, V^) of Dn and two couples of vectors (e, /) and 
(e, (/>) (respectively in ^4- and Vr) such that (/9,r,e,/(5') = VT,e,4>{g)-, for all g G G^, with a suitable 
choice of group G^ . 

With notation as in |Serl 5.3], let G^ = G„, be the cyclic group of order n. It is of index 2 in 
Dn and we have an exact sequence of finite groups: 

l^Gn^Dn^ Z/2Z ^ 1 . 

We fix the trivial left coset representative a = 1 (with respect to the quotient Dn/Gn)- If hi 
and /i2 are two distinct integers such that hi = — /i2 (modn), then the representations p^^ and 
p^^ given in the canonical basis of by 

h i _ f ^'"''^ 
P'ir )-[ Q ^-h,k 

(where, as in loc. cit., r is a generator for G„, ^ A; ^ n — 1 and uj = exp(2i7r/n)), are irreducible 
of degree 2 and are not isomorphic. A straightforward computation shows that the canonical 
basis (e, /) of is in fact an orthonormal basis of the representation space V^h^ , i = 1, 2, with 
respect to the Z)„-invariant inner product ( ; constructed from the canonical scalar product 
on by averaging over Dn- Then we have, for all g £ Cn, 

since, for all ^ /c ^ n — 1, the equality oj^'^'' = uj~^'^^ holds. 
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Via that example, we also see that there do exist functions '^■K,e,f that vanish identically on a 
whole left coset of G = D„ with respect to = Cn- Indeed, we have, for ^ /c ^ n — 1, 

i.e. ^pH^ejia) = for all g e Cn- 

A common feature in the two lemmas above is that each individual sum W{ip, ip') involves two 
(a priori) distinct representations (of two a priori distinct groups). In order to estimate those 
sums, it will be convenient to rewrite them in such a way that a single group representation 
appears for each of the W{Lp,ip'). For that purpose, we need to introduce additional notation: 
for I and I' two elements of A, let Gn^^i be the group: 



Gf 



Ge X Ge if i ^ i' , 
Gf , otherwise . 



Now, for vr (resp. r) an irreducible representation of Gg (resp. Gc), we define the represen- 
tation of Gf fi: 



[7r,r] 



TT K r if £ f 

vr (g) r , otherwise . 



where "Kl" (resp. "(gi") denotes the external (resp. inner) tensor product of representations. 
With such notation, we can give the statement of |KoSieve[ Lemma 3.4] which is useful in the 
proofs of Propositions [6] and [71 

Lemma 24. Let £, £' in A, vr (resp. t) a non trivial irreducible representation of Gf (resp. of 
Gf). The multiplicity of the trivial representation in the restriction o/[7r,f] to G^ is equal to 

zero if {£,tt) ^ {£',t), and is equal to \TJ\ if {£,tt) = {£',t). 

If £ ^ £', it is well known that the family ([7r,r]) of representations of Gf^f, with vr (resp. r) 
running over a system of representatives of irreducible representations of Gf (resp. Gf) forms 
itself a system of representatives for the irreducible representations of Gf^f . 

The above notation and the sieving context we would like to work with, suggest us to combine 
the maps pf and pf (assuming £ and £' are distinct elements of A) in a single map 

Pe,e' '■ G Gf^f 

9 ^ {Pe{9),Pe'{9)) , 

which is nothing but the product map from G to x Gf . 

Now we claim that the exponential sums (fT2]) can be rewritten, according to the sieve setting 
considered, in one of the following forms: 

• in the case of the conjugacy coset sieve, we have 

(14) W{ip^,ipr) = —=^=BiTri[Tr,f]pf^f{Xk)), 

y l-^ mil nl 

with notation as in Lemma [22l 

• in the case of the non-conjugacy coset sieve, we have 



(15) M^(v..,ej,^.,.,^) = J fZ'!f!^Z^,l ^^< [^,r]iPi,i'(Xk))e;f >[., 



with notation as in Lemma [231 and where e = e, f = f ' 



Both facts are a direct application of |KoSieve[ Lemma 2.11]. 
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4.3. Self-contained statements. We finish this appendix by giving self-contained statements 
(i.e. using no new terminology) for Lemmas [22] and [23] in order to make it possible for the reader 
to follow the whole proof of Theorem [1] without having to get too much involved (at least for 
a first reading of the paper) in the details of the sieve. To begin with, we give the following 
self-contained version of Lemma [22] (this is [KoSievel Prop. 3.7]): 

Proposition 25. Let G be a group, a normal subgroup of G with abelian quotient T; denote 
d : G — > r the quotient map. Let A be a subset of the rational primes and let p£ : G ^ Gi, for 
I €z A, be a family of surjective homomorphisms onto finite groups. Denote G^ = pi{G^). Let 
a & T be fixed, Y = d~^{a) C G and = pe{Y). Let (^,11, P) be a probability space and X a 
random variable with values in Y . For any 

i & A let Hi be a set of representatives of the set of irreducible representations of Gi modulo 
equality restricted to containing the constant function 1. Moreover, let 11^ = 11^ \ {1} and 
rj be the set of characters ip ofTi = Gi/Gf such that vr ~ vr for a representation vr of Gi. 

Let C* be a finite subset of A. Then, for any conjugacy invariant subsets Qe C Yi for i € C* , 
we have 

P{pe{X) i G^, for all £ G £*) ^ t\R~^ 
where A is the smallest non-negative real number such that 

|E(/3.Trvr(M^)))|'^AE(|/3|2) 
for all square-integrable functions [5 G L^(^',P), and 

ie,i 



In addition, we have 



\Gi\-m- 



(16) 5Z E 

^ I'&c ren*, 

with 

(17) W{t^,t) = ^ ^ E(Tr^(p,(X))TVr(p,K^))) = , ^ E(Tr[^, f](p,,,K^))) , 



"p7r||"pT| / I T7I" I I "PT I 

using the notation pi^e for the product map p£ x pi' : G ^ Gg x G^i if i ^ ^ and pi^i' = pi 
otherwise, and [vr,f] = vr (gi r for the (internal or external, depending on whether 1 = 1' or not) 
tensor product of the representations vr and f. 

The analogue self-contained statement in the non-conjugacy coset sieve setting is the following 
reformulation of Lemma [22 

Proposition 26. Let {G,G3 , A, (pe), (Gi), (G^^)), (^',S,P) and a,Y,{Y(>),X be as in Propo- 
sition [13 . Moreover, for each i € A and each finite dimensional irreducible representation 
IT £ lrr:{Gi) (the set of isomorphism classes of such irreducible representations), let 

B^ = {el,...,ef^n 

be an orthonormal basis of the space of it with respect to a G^-invariant inner product { ; )7r. For 
the set of triples {(vr, e, /) | vr € Irr(G£), e, / G B,^}, we denote by a set of representatives for 
the equivalence relation: 

(tt, e, /) ~ (r, e, (f>) if {Tr{g)e; f)„ = {T{g)e; <j))r , for all g £ G^ 

such that (l,e, e) € (where 1 denotes the trivial representation and e is a basis for the 1- 
dimensional space attached to it) and such that there is no (vr, e, /) € satisfying {7r{g)e; f)-,^ = 
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for all g €z . Let n| = \ {(l,e,e)} and let £* be a finite subset of A. Then, for any 
subsets @i <ZYg for £ G C* , we have 

P{pi{X) ^ e^, for £ G £*) ^ l^H-^ 

where A is the smallest non-negative real number such that 

E \/di^|E(/3 • {7r{pe{X))e; f))f ^ AB{\P\^) 

(7r,e,/)gn* 

for all square-integrable functions P G L^(^',P), where 

rr V IQ^I 

i^JGA-\@i\- 

Moreover we have 



(18) A < max max V ^ |17((7r, e, /), (r, </.))! , 

where, with the same notations as in 



(dim7r)(dimr) 



(19) W{{7T,eJ),iT,E,<P)) = ) \, ^^J^ E((^(p,(^))e; /). (r(pKX))e; </,),) 

y |r^ ' ' ||r^, ' ' I 

/ (dim7r)(dimr) „ , ^ 

= W L.,,, ® £); (/ ^ 0))[.,f]) • 

y l-l m 1 1"'- n I 

with Y^^'^''^^ denoting the set of characters x of Ti such that 

{T^{g)e; f)TT ■ xia) = {'^{g)e-; f)n ■ 

Remark. In our description of coset sieves, we have restricted ourselves to sieve supports contain- 
ing only prime numbers. Nevertheless as suggested by the discussions preceding and following 
Lemma [211 we could quite easily extend our sieve method to a framework in which we would use 
squarefree integers (and not only primes) as a sieve support. As described in [KoSievei Chap. 
3], going from a prime sieve support to a "squarefree" sieve support can be done naturally by 
extending a few of the definitions we have given in this appendix by multiplicativity. Although 
using that extended sieve support would surely yield better estimates in Theorem [H we prefer 
working only with a prime sieve support, so that we avoid the use of additional notation. How- 
ever, for the proof of Proposition [71 it is convenient to use objects defined by multiplicativity 
from two (not more) primes in A. So, for ^ 7^ i' two such primes, let 

Yg^e' = Yex Yp , 

on which we have the product density ve,^e{v-,v') = ^t{v)^i'{y'), so that it makes sense to 
speak about the space (Y^^^/ , i/^^^/ ) . It is straightforward to check that if B/^ (resp. B^i) 
is an orthonormal basis of ^^(y^,^^) (resp. of Lp'iY^i ,1/(^1))^ the family of functions defined 
by {ViV') G I— ^ viy)f'{y')7 where (p G Bg and ip' G S^/, forms an orthonormal basis of 
L'^iYe/'ji^i/'). Note finally, that, to unify all the possible cases, we can extend the above defini- 
tions to the case £ = i' hy defining Y^^^' = Y^, = lyg and B^^i' = Bg. 
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